MonitorWare Knowledge Base

by **hkspvt** » Tue Jul 01, 2008 10:29 pm

Hi again.

Setting a queue other than Direct (LinkedList, FixedArray, or Disk) causes a core dump on my machine. FreeBSD 7.0, rsyslog 3.17.4, compiled with the --enable-debug option. Here's what my working rsyslog.conf file looks like:

Code: Select all: $ActionQueueType Direct *.* /var/log/debug.log

Changing Direct to any of the others crashes rsyslog almost immediately upon starting. Here's the output of an interactive, debugging run when LinkedList was configured:

Code: Select all: qa-freebsd# /usr/local/sbin/rsyslogd -c3 -dn -f /usr/local/etc/rsyslog.conf 7647.094113627:main thread: Writing pidfile /var/run/rsyslogd.pid. 7647.096137068:main thread: rsyslog 3.17.4 - called init() 7647.096139024:main thread: Unloading non-static modules. 7647.096140980:main thread: module lmnet NOT unloaded because it still has a refcount of 3 7647.096580980:main thread: module lmtcpclt NOT unloaded because it still has a refcount of 1 7647.096582935:main thread: modules still loaded after module.UnloadAndDestructAll: 7647.096584891:main thread: printing users of loadable module lmnet, refcount 3, ptr 0x28213080, type 2 7647.096587126:main thread: module lmnet is currently in use by file omfwd.c 7647.096589081:main thread: module lmnet is currently in use by file syslogd.c 7647.096591037:main thread: module lmnet is currently in use by file conf.c 7647.096592992:main thread: printing users of loadable module builtin-file, refcount 0, ptr 0x282130e0, type 1 7647.096594948:main thread: printing users of loadable module lmtcpclt, refcount 1, ptr 0x282131a0, type 2 7647.096596903:main thread: module lmtcpclt is currently in use by file omfwd.c 7647.096598859:main thread: printing users of loadable module builtin-fwd, refcount 0, ptr 0x28213140, type 1 7647.096601094:main thread: printing users of loadable module builtin-shell, refcount 0, ptr 0x28213200, type 1 7647.096605005:main thread: printing users of loadable module builtin-discard, refcount 0, ptr 0x28213260, type 1 7647.096606960:main thread: printing users of loadable module builtin-usrmsg, refcount 0, ptr 0x282132c0, type 1 7647.096608916:main thread: Clearing templates. 7647.098056027:main thread: cfline: '$ActionQueueType LinkedList' 7647.098089830:main thread: action queue type set to LINKEDLIST 7647.098091786:main thread: cfline: '*.* /var/log/debug.log' 7647.098163304:main thread: - traditional PRI filter 7647.098165259:main thread: symbolic name: * ==> 255 7647.098692421:main thread: tried selector action for builtin-file: 0 7647.098694377:main thread: Module builtin-file processed this config line. 7647.099180472:main thread: template: 'RSYSLOG_FileFormat' assigned 7647.099557615:main thread: action 1 queue: save on shutdown 1, max disk space allowed 0 7647.099615723:main thread: action 1 queue: is NOT disk-assisted 7647.099617678:main thread: action 1 queue: type 1, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 7647.099696459:main thread: action 1 queue:Reg: finalizing construction of worker thread pool 7647.099698415:main thread: action 1 queue:Reg/w0: finalizing construction of worker instance data 7647.099700370:main thread: action 1 queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 7647.099702326:main thread: action 1 queue:Reg: high activity - starting 1 additional worker thread(s). 7647.099704561:main thread: action 1 queue:Reg/w0: receiving command 2 7647.100652447:main thread: action 1 queue:Reg: started with state 0, num workers now 1 7647.102434796:main thread: Action 0x2821f0b0: queue 0x28224260 created 7647.102602415:main thread: selector line successfully processed 7647.102674212:main thread: main queue: is NOT disk-assisted 7647.102676168:main thread: main queue: type 0, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 7647.103064485:main thread: main queue:Reg: finalizing construction of worker thread pool 7647.103066441:main thread: main queue:Reg/w0: finalizing construction of worker instance data 7647.103068396:main thread: main queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 7647.103070352:main thread: main queue:Reg: high activity - starting 1 additional worker thread(s). 7647.103072307:main thread: main queue:Reg/w0: receiving command 2 7647.103651711:main thread: main queue:Reg: started with state 0, num workers now 1 7647.103705628:28201200: action 1 queue:Reg/w0: receiving command 4 7647.105066136:main thread: Main processing queue is initialized and running 7647.105068371:main thread: Active selectors: 7647.105139609:main thread: Selector 1: 7647.105141565:main thread: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Actions: 7647.106559343:main thread: builtin-file: /var/log/debug.log Instance data: 0x28224140 7647.106632536:main thread: RepeatedMsgReduction: 0 7647.106634492:main thread: Resume Interval: 30 7647.106636448:main thread: Suspended: 0 7647.107060244:main thread: Disabled: 0 7647.107062479:main thread: Exec only when previous is suspended: 0 7647.107064435:main thread: 7647.107066390:main thread: 7647.107068346:main thread: 7647.107070301:main thread: 7647.107074213:main thread: Template: Name='RSYSLOG_SyslogProtocol23Format' 7647.107609756:main thread: Entry(28205680): type 1, (CONSTANT), value: '<' 7647.107664790:main thread: Entry(282056c0): type 2, (FIELD), value: 'pri' 7647.107720105:main thread: Entry(28205700): type 1, (CONSTANT), value: '>1 ' 7647.108103394:main thread: Entry(28205740): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 7647.108172676:main thread: Entry(28205780): type 1, (CONSTANT), value: ' ' 7647.108558200:main thread: Entry(282057c0): type 2, (FIELD), value: 'hostname' 7647.108617705:main thread: Entry(28205800): type 1, (CONSTANT), value: ' ' 7647.108673019:main thread: Entry(28205840): type 2, (FIELD), value: 'app-name' 7647.109060778:main thread: Entry(28205880): type 1, (CONSTANT), value: ' ' 7647.109116092:main thread: Entry(282058c0): type 2, (FIELD), value: 'procid' 7647.109170848:main thread: Entry(28205900): type 1, (CONSTANT), value: ' ' 7647.109557489:main thread: Entry(28205940): type 2, (FIELD), value: 'msgid' 7647.109614200:main thread: Entry(28205980): type 1, (CONSTANT), value: ' ' 7647.109668956:main thread: Entry(282059c0): type 2, (FIELD), value: 'structured-data' 7647.110057273:main thread: Entry(28205a00): type 1, (CONSTANT), value: ' ' 7647.110564600:main thread: Entry(28205a40): type 2, (FIELD), value: 'msg' 7647.110648689:main thread: Entry(28205a80): type 1, (CONSTANT), value: ' ' 7647.110716016:main thread: Template: Name='RSYSLOG_FileFormat' 7647.111086454:main thread: Entry(28205ac0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 7647.111155458:main thread: Entry(28205b00): type 1, (CONSTANT), value: ' ' 7647.111210213:main thread: Entry(28205b40): type 2, (FIELD), value: 'hostname' 7647.111625350:main thread: Entry(28205b80): type 1, (CONSTANT), value: ' ' 7647.111680943:main thread: Entry(28205bc0): type 2, (FIELD), value: 'syslogtag' 7647.112068423:main thread: Entry(28205c00): type 2, (FIELD), value: 'msg' [drop last LF in msg] 7647.112137985:main thread: Entry(28205c40): type 1, (CONSTANT), value: ' ' 7647.112204474:main thread: Template: Name='RSYSLOG_TraditionalFileFormat' 7647.112208385:main thread: Entry(28205c80): type 2, (FIELD), value: 'timestamp' 7647.112640004:main thread: Entry(28205cc0): type 1, (CONSTANT), value: ' ' 7647.112695039:main thread: Entry(28205d00): type 2, (FIELD), value: 'hostname' 7647.113086429:main thread: Entry(28205d40): type 1, (CONSTANT), value: ' ' 7647.113141743:main thread: Entry(28205d80): type 2, (FIELD), value: 'syslogtag' 7647.113196499:main thread: Entry(28205dc0): type 2, (FIELD), value: 'msg' [drop last LF in msg] 7647.113597109:main thread: Entry(28205e00): type 1, (CONSTANT), value: ' ' 7647.113663597:main thread: Template: Name=' WallFmt' 7647.113704106:main thread: Entry(28205e40): type 1, (CONSTANT), value: ' Message from syslogd@' 7647.114110302:main thread: Entry(28205e80): type 2, (FIELD), value: 'hostname' 7647.114165617:main thread: Entry(28205ec0): type 1, (CONSTANT), value: ' at ' 7647.114555610:main thread: Entry(28205f00): type 2, (FIELD), value: 'timegenerated' 7647.114612322:main thread: Entry(28205f40): type 1, (CONSTANT), value: ' ... ' 7647.114679649:main thread: Entry(28205f80): type 2, (FIELD), value: 'syslogtag' 7647.115219661:main thread: Entry(28205fc0): type 2, (FIELD), value: 'msg' 7647.115633680:main thread: Entry(2821d040): type 1, (CONSTANT), value: ' ' 7647.115709947:main thread: Template: Name='RSYSLOG_ForwardFormat' 7647.116156372:main thread: Entry(2821d080): type 1, (CONSTANT), value: '<' 7647.116211687:main thread: Entry(2821d0c0): type 2, (FIELD), value: 'pri' 7647.116561172:main thread: Entry(2821d100): type 1, (CONSTANT), value: '>' 7647.116618163:main thread: Entry(2821d140): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 7647.116689960:main thread: Entry(2821d180): type 1, (CONSTANT), value: ' ' 7647.117077160:main thread: Entry(2821d1c0): type 2, (FIELD), value: 'hostname' 7647.117132195:main thread: Entry(2821d200): type 1, (CONSTANT), value: ' ' 7647.117186950:main thread: Entry(2821d240): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 7647.117559344:main thread: Entry(2821d280): type 2, (FIELD), value: 'msg' 7647.117643712:main thread: Template: Name='RSYSLOG_TraditionalForwardFormat' 7647.117684500:main thread: Entry(2821d2c0): type 1, (CONSTANT), value: '<' 7647.118071700:main thread: Entry(2821d300): type 2, (FIELD), value: 'pri' 7647.118129249:main thread: Entry(2821d340): type 1, (CONSTANT), value: '>' 7647.118184563:main thread: Entry(2821d380): type 2, (FIELD), value: 'timestamp' 7647.118559750:main thread: Entry(2821d3c0): type 1, (CONSTANT), value: ' ' 7647.118638811:main thread: Entry(2821d400): type 2, (FIELD), value: 'hostname' 7647.118694405:main thread: Entry(2821d440): type 1, (CONSTANT), value: ' ' 7647.119082163:main thread: Entry(2821d480): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 7647.119152005:main thread: Entry(2821d4c0): type 2, (FIELD), value: 'msg' 7647.119207039:main thread: Template: Name=' StdUsrMsgFmt' 7647.119210951:main thread: Entry(2821d500): type 1, (CONSTANT), value: ' ' 7647.119642849:main thread: Entry(2821d540): type 2, (FIELD), value: 'syslogtag' 7647.120720360:main thread: Entry(2821d580): type 2, (FIELD), value: 'msg' 7647.121070684:main thread: Entry(2821d5c0): type 1, (CONSTANT), value: ' ' 7647.121139408:main thread: Template: Name=' StdDBFmt' [SQL-Format (MySQL)] 7647.121203103:main thread: Entry(2821d600): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 7647.121207014:main thread: Entry(2821d640): type 2, (FIELD), value: 'msg' 7647.121653161:main thread: Entry(2821d680): type 1, (CONSTANT), value: '', ' 7647.121708195:main thread: Entry(2821d6c0): type 2, (FIELD), value: 'syslogfacility' 7647.122089529:main thread: Entry(2821d700): type 1, (CONSTANT), value: ', '' 7647.122144564:main thread: Entry(2821d740): type 2, (FIELD), value: 'hostname' 7647.122199040:main thread: Entry(2821d780): type 1, (CONSTANT), value: '', ' 7647.122202951:main thread: Entry(2821d7c0): type 2, (FIELD), value: 'syslogpriority' 7647.122644627:main thread: Entry(2821d800): type 1, (CONSTANT), value: ', '' 7647.122699383:main thread: Entry(2821d840): type 2, (FIELD), value: 'timereported' [Format as MySQL-Date] 7647.123101389:main thread: Entry(2821d880): type 1, (CONSTANT), value: '', '' 7647.123156424:main thread: Entry(2821d8c0): type 2, (FIELD), value: 'timegenerated' [Format as MySQL-Date] 7647.123557592:main thread: Entry(2821d900): type 1, (CONSTANT), value: '', ' 7647.123614304:main thread: Entry(2821d940): type 2, (FIELD), value: 'iut' 7647.123672132:main thread: Entry(2821d980): type 1, (CONSTANT), value: ', '' 7647.124060729:main thread: Entry(2821d9c0): type 2, (FIELD), value: 'syslogtag' 7647.124115764:main thread: Entry(2821da00): type 1, (CONSTANT), value: '')' 7647.124180856:main thread: Template: Name=' StdPgSQLFmt' [SQL-Format (standard SQL)] 7647.124554367:main thread: Entry(2821da40): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 7647.125595282:main thread: Entry(2821da80): type 2, (FIELD), value: 'msg' 7647.125655345:main thread: Entry(2821dac0): type 1, (CONSTANT), value: '', ' 7647.125710939:main thread: Entry(2821db00): type 2, (FIELD), value: 'syslogfacility' 7647.126097860:main thread: Entry(2821db40): type 1, (CONSTANT), value: ', '' 7647.126153174:main thread: Entry(2821db80): type 2, (FIELD), value: 'hostname' 7647.126208209:main thread: Entry(2821dbc0): type 1, (CONSTANT), value: '', ' 7647.126599599:main thread: Entry(2821dc00): type 2, (FIELD), value: 'syslogpriority' 7647.126654634:main thread: Entry(2821dc40): type 1, (CONSTANT), value: ', '' 7647.126709390:main thread: Entry(2821dc80): type 2, (FIELD), value: 'timereported' [Format as PgSQL-Date] 7647.127125364:main thread: Entry(2821dcc0): type 1, (CONSTANT), value: '', '' 7647.127564806:main thread: Entry(2821dd00): type 2, (FIELD), value: 'timegenerated' [Format as PgSQL-Date] 7647.127652806:main thread: Entry(2821dd40): type 1, (CONSTANT), value: '', ' 7647.127707841:main thread: Entry(2821dd80): type 2, (FIELD), value: 'iut' 7647.128147841:main thread: Entry(2821ddc0): type 1, (CONSTANT), value: ', '' 7647.128562139:main thread: Entry(2821de00): type 2, (FIELD), value: 'syslogtag' 7647.128622761:main thread: Entry(2821de40): type 1, (CONSTANT), value: '')' 7647.128678914:main thread: Loaded Module: Name='lmnet', IFVersion=4, type=library module. 7647.129079803:main thread: Entry points: 7647.129081758:main thread: queryEtryPt: 0x281d7500 7647.129083714:main thread: doAction: 0x0 7647.129085669:main thread: parseSelectorAct: 0x0 7647.129087625:main thread: dbgPrintInstInfo: 0x0 7647.129089860:main thread: freeInstance: 0x0 7647.129093771:main thread: 7647.129095727:main thread: Loaded Module: Name='builtin-file', IFVersion=4, type=output module. 7647.129669263:main thread: Entry points: 7647.129671219:main thread: queryEtryPt: 0x806fad0 7647.129673174:main thread: doAction: 0x8070900 7647.131129225:main thread: parseSelectorAct: 0x8070030 7647.131131181:main thread: dbgPrintInstInfo: 0x8070710 7647.131133136:main thread: freeInstance: 0x806ffb0 7647.131135092:main thread: 7647.131139003:main thread: Loaded Module: Name='lmtcpclt', IFVersion=4, type=library module. 7647.131645213:main thread: Entry points: 7647.131647168:main thread: queryEtryPt: 0x281dbd60 7647.131649403:main thread: doAction: 0x0 7647.131651359:main thread: parseSelectorAct: 0x0 7647.131653314:main thread: dbgPrintInstInfo: 0x0 7647.131655270:main thread: freeInstance: 0x0 7647.131657225:main thread: 7647.131659181:main thread: Loaded Module: Name='builtin-fwd', IFVersion=4, type=output module. 7647.132557060:main thread: Entry points: 7647.132559016:main thread: queryEtryPt: 0x806e7a0 7647.132560971:main thread: doAction: 0x806ecf0 7647.132562927:main thread: parseSelectorAct: 0x806f0b0 7647.132564882:main thread: dbgPrintInstInfo: 0x806ecd0 7647.132566838:main thread: freeInstance: 0x806ec30 7647.132569073:main thread: 7647.132571029:main thread: Loaded Module: Name='builtin-shell', IFVersion=4, type=output module. 7647.133174457:main thread: Entry points: 7647.133176413:main thread: queryEtryPt: 0x806d720 7647.133180324:main thread: doAction: 0x806da80 7647.133182279:main thread: parseSelectorAct: 0x806d8b0 7647.133184235:main thread: dbgPrintInstInfo: 0x806da60 7647.133186470:main thread: freeInstance: 0x806d890 7647.133188426:main thread: 7647.133190381:main thread: Loaded Module: Name='builtin-discard', IFVersion=4, type=output module. 7647.134101111:main thread: Entry points: 7647.134103067:main thread: queryEtryPt: 0x8070fa0 7647.134105022:main thread: doAction: 0x8071110 7647.134106978:main thread: parseSelectorAct: 0x8071150 7647.134109213:main thread: dbgPrintInstInfo: 0x8070e70 7647.134113124:main thread: freeInstance: 0x8071130 7647.134115080:main thread: 7647.135635385:main thread: Loaded Module: Name='builtin-usrmsg', IFVersion=4, type=output module. 7647.135707181:main thread: Entry points: 7647.135709416:main thread: queryEtryPt: 0x806dc40 7647.135711372:main thread: doAction: 0x806e4b0 7647.135713327:main thread: parseSelectorAct: 0x806de20 7647.135715283:main thread: dbgPrintInstInfo: 0x806ddb0 7647.135717238:main thread: freeInstance: 0x806de00 7647.135719194:main thread: 7647.135723385:main thread: Sytem Line Configuration Commands: 7647.135725340:main thread: Command 'dynafilecachesize': 7647.135727296:main thread: type : 6 7647.135729251:main thread: pData: 0x0 7647.135731207:main thread: Hdlr : 0x806fc40 7647.135733442:main thread: Owner: 0x806fe80 7647.135735397:main thread: 7647.135737353:main thread: Command 'dirowner': 7647.135739308:main thread: type : 2 7647.135741264:main thread: pData: 0x8084d90 7647.135743219:main thread: Hdlr : 0x0 7647.135745454:main thread: Owner: 0x806fe80 7647.135749365:main thread: 7647.135751321:main thread: Command 'dirgroup': 7647.135753276:main thread: type : 3 7647.135755232:main thread: pData: 0x8084d94 7647.135757188:main thread: Hdlr : 0x0 7647.135759423:main thread: Owner: 0x806fe80 7647.135761378:main thread: 7647.135763334:main thread: Command 'fileowner': 7647.135765289:main thread: type : 2 7647.135767245:main thread: pData: 0x8084d88 7647.135769200:main thread: Hdlr : 0x0 7647.135771435:main thread: Owner: 0x806fe80 7647.135775347:main thread: 7647.135777302:main thread: Command 'filegroup': 7647.135779258:main thread: type : 3 7647.135781213:main thread: pData: 0x8084d8c 7647.135783448:main thread: Hdlr : 0x0 7647.135785404:main thread: Owner: 0x806fe80 7647.135787359:main thread: 7647.135789315:main thread: Command 'dircreatemode': 7647.135791270:main thread: type : 5 7647.135793226:main thread: pData: 0x8082a50 7647.135795181:main thread: Hdlr : 0x0 7647.135797416:main thread: Owner: 0x806fe80 7647.140563385:main thread: 7647.140565341:main thread: Command 'filecreatemode': 7647.140567296:main thread: type : 5 7647.140569252:main thread: pData: 0x8082a54 7647.140571207:main thread: Hdlr : 0x0 7647.140573442:main thread: Owner: 0x806fe80 7647.140575398:main thread: 7647.140577353:main thread: Command 'createdirs': 7647.140579309:main thread: type : 4 7647.140581265:main thread: pData: 0x8084d98 7647.140583220:main thread: Hdlr : 0x0 7647.140585455:main thread: Owner: 0x806fe80 7647.140589366:main thread: 7647.140591322:main thread: Command 'failonchownfailure': 7647.140593277:main thread: type : 4 7647.140595233:main thread: pData: 0x8084d84 7647.140597188:main thread: Hdlr : 0x0 7647.140599423:main thread: Owner: 0x806fe80 7647.140601379:main thread: 7647.140603334:main thread: Command 'actionfileenablesync': 7647.140605290:main thread: type : 4 7647.140607245:main thread: pData: 0x8084d78 7647.140609201:main thread: Hdlr : 0x0 7647.140611436:main thread: Owner: 0x806fe80 7647.140615347:main thread: 7647.140617303:main thread: Command 'actionfiledefaulttemplate': 7647.140619258:main thread: type : 11 7647.140621214:main thread: pData: 0x8084d7c 7647.140623449:main thread: Hdlr : 0x0 7647.140625404:main thread: Owner: 0x0 7647.140627360:main thread: 7647.140629315:main thread: Command 'resetconfigvariables': 7647.140631271:main thread: type : 1 7647.140633226:main thread: pData: 0x0 7647.140635461:main thread: Hdlr : 0x806fea0 7647.140637417:main thread: Owner: 0x806fe80 7647.140641328:main thread: 7647.140643283:main thread: type : 1 7647.140645239:main thread: pData: 0x0 7647.140647195:main thread: Hdlr : 0x806eb60 7647.140649429:main thread: Owner: 0x806eb90 7647.140651385:main thread: 7647.140653341:main thread: type : 1 7647.140655296:main thread: pData: 0x0 7647.140657252:main thread: Hdlr : 0x804f8c0 7647.140659207:main thread: Owner: 0x0 7647.140661442:main thread: 7647.145187157:main thread: Command 'actionforwarddefaulttemplate': 7647.145189113:main thread: type : 11 7647.145191068:main thread: pData: 0x8084d70 7647.145195259:main thread: Hdlr : 0x0 7647.145197214:main thread: Owner: 0x0 7647.145199170:main thread: 7647.145201125:main thread: Command 'workdirectory': 7647.145203081:main thread: type : 11 7647.145205316:main thread: pData: 0x8083c98 7647.145207271:main thread: Hdlr : 0x0 7647.145209227:main thread: Owner: 0x0 7647.145211183:main thread: 7647.145213138:main thread: Command 'actionresumeretrycount': 7647.145215094:main thread: type : 6 7647.145217049:main thread: pData: 0x8084dd0 7647.145221240:main thread: Hdlr : 0x0 7647.145223195:main thread: Owner: 0x0 7647.145225151:main thread: 7647.145227106:main thread: Command 'mainmsgqueuefilename': 7647.145229062:main thread: type : 11 7647.145231297:main thread: pData: 0x8083cc4 7647.145233252:main thread: Hdlr : 0x0 7647.145235208:main thread: Owner: 0x0 7647.145237163:main thread: 7647.145239119:main thread: Command 'mainmsgqueuesize': 7647.145241075:main thread: type : 6 7647.145243309:main thread: pData: 0x8081300 7647.145247221:main thread: Hdlr : 0x0 7647.145249176:main thread: Owner: 0x0 7647.145251132:main thread: 7647.147708986:main thread: Command 'mainmsgqueuehighwatermark': 7647.147711221:main thread: type : 6 7647.147713176:main thread: pData: 0x8081304 7647.147715132:main thread: Hdlr : 0x0 7647.147717087:main thread: Owner: 0x0 7647.147719043:main thread: 7647.147720999:main thread: Command 'mainmsgqueuelowwatermark': 7647.147725189:main thread: type : 6 7647.147727145:main thread: pData: 0x8081308 7647.147729100:main thread: Hdlr : 0x0 7647.147731056:main thread: Owner: 0x0 7647.147733011:main thread: 7647.147735246:main thread: Command 'mainmsgqueuediscardmark': 7647.147737202:main thread: type : 6 7647.147739158:main thread: pData: 0x808130c 7647.147741113:main thread: Hdlr : 0x0 7647.110130467:action 1 queue:Reg/w0: action 1 queue: entering rate limiter 7647.152669114:28201300: main queue:Reg/w0: receiving command 4 7647.153090955:main thread: Owner: 0x0 7647.153092911:main thread: 7647.153094866:main thread: Command 'mainmsgqueuediscardseverity': 7647.153096822:main thread: type : 10 7647.153098777:main thread: pData: 0x8081310 7647.153102968:main thread: Hdlr : 0x0 7647.153104923:main thread: Owner: 0x0 7647.153106879:main thread: 7647.153108834:main thread: Command 'mainmsgqueuecheckpointinterval': 7647.153110790:main thread: type : 6 7647.153112745:main thread: pData: 0x8083cc8 7647.153114980:main thread: Hdlr : 0x0 7647.153116936:main thread: Owner: 0x0 7647.153118891:main thread: 7647.153120847:main thread: Command 'mainmsgqueuetype': 7647.153122803:main thread: type : 11 7647.153124758:main thread: pData: 0x0 7647.153126993:main thread: Hdlr : 0x804ff50 7647.153130904:main thread: Owner: 0x0 7647.153132860:main thread: 7647.153134815:main thread: Command 'mainmsgqueueworkerthreads': 7647.153136771:main thread: type : 6 7647.153138726:main thread: pData: 0x8081314 7647.153140961:main thread: Hdlr : 0x0 7647.153142917:main thread: Owner: 0x0 7647.153144872:main thread: 7647.153146828:main thread: Command 'mainmsgqueuetimeoutshutdown': 7647.153148784:main thread: type : 6 7647.153150739:main thread: pData: 0x8083ccc 7647.153154930:main thread: Hdlr : 0x0 7647.153156885:main thread: Owner: 0x0 7647.153158841:main thread: 7647.153160796:main thread: Command 'mainmsgqueuetimeoutactioncompletion': 7647.153162752:main thread: type : 6 7647.153164987:main thread: pData: 0x8081318 7647.153166942:main thread: Hdlr : 0x0 7647.153168898:main thread: Owner: 0x0 7647.153170853:main thread: 7647.153172809:main thread: Command 'mainmsgqueuetimeoutenqueue': 7647.153174764:main thread: type : 6 7647.153176999:main thread: pData: 0x808131c 7647.153180910:main thread: Hdlr : 0x0 7647.153182866:main thread: Owner: 0x0 7647.153184822:main thread: 7647.153186777:main thread: Command 'mainmsgqueueworketimeoutrthreadshutdown': 7647.157070511:main thread: type : 6 7647.157072467:main thread: pData: 0x8081320 7647.157074422:main thread: Hdlr : 0x0 7647.157076378:main thread: Owner: 0x0 7647.157078333:main thread: 7647.157080289:main thread: Command 'mainmsgqueuedequeueslowdown': 7647.157084479:main thread: type : 6 7647.157086435:main thread: pData: 0x8083cd0 7647.157088390:main thread: Hdlr : 0x0 7647.157090346:main thread: Owner: 0x0 7647.157092302:main thread: 7647.157094536:main thread: Command 'mainmsgqueueworkerthreadminimummessages': 7647.157096492:main thread: type : 6 7647.157098448:main thread: pData: 0x8081324 7647.157100403:main thread: Hdlr : 0x0 7647.157102359:main thread: Owner: 0x0 7647.157104314:main thread: 7647.157106270:main thread: Command 'mainmsgqueuemaxfilesize': 7647.157110460:main thread: type : 7 7647.157112416:main thread: pData: 0x8081328 7647.157114371:main thread: Hdlr : 0x0 7647.157116327:main thread: Owner: 0x0 7647.157118282:main thread: 7647.157120517:main thread: Command 'mainmsgqueuemaxdiskspace': 7647.157122473:main thread: type : 7 7647.157124429:main thread: pData: 0x8083cd8 7647.157126384:main thread: Hdlr : 0x0 7647.157128340:main thread: Owner: 0x0 7647.157130295:main thread: 7647.157132530:main thread: Command 'mainmsgqueuesaveonshutdown': 7647.157136441:main thread: type : 4 7647.157138397:main thread: pData: 0x8081330 7647.157140352:main thread: Hdlr : 0x0 7647.157142308:main thread: Owner: 0x0 7647.157144543:main thread: 7647.157146498:main thread: Command 'mainmsgqueuedequeuetimebegin': 7647.157148454:main thread: type : 6 7647.157150409:main thread: pData: 0x8083ce0 7647.157152365:main thread: Hdlr : 0x0 7647.157154320:main thread: Owner: 0x0 7647.157156276:main thread: 7647.157158511:main thread: Command 'mainmsgqueuedequeuetimeend': 7647.157160467:main thread: type : 6 7647.157164378:main thread: pData: 0x8081334 7647.157166333:main thread: Hdlr : 0x0 7647.160648899:main queue:Reg/w0: main queue: entering rate limiter 7647.163663528:action 1 queue:Reg/w0: action 1 queue:Reg/w0: worker IDLE, waiting for work. 7647.163712417:main thread: Owner: 0x0 7647.163714651:main thread: 7647.163716607:main thread: Command 'repeatedmsgreduction': 7647.163718563:main thread: type : 4 7647.163720518:main thread: pData: 0x8084dfc 7647.163724429:main thread: Hdlr : 0x0 7647.163726385:main thread: Owner: 0x0 7647.163728620:main thread: 7647.163730575:main thread: Command 'actionexeconlywhenpreviousissuspended': 7647.163732531:main thread: type : 4 7647.163734487:main thread: pData: 0x8084e00 7647.163736442:main thread: Hdlr : 0x0 7647.163738398:main thread: Owner: 0x0 7647.163740633:main thread: 7647.165187744:main thread: Command 'actionexeconlyonceeveryinterval': 7647.165189699:main thread: type : 6 7647.165593941:main thread: pData: 0x8083c94 7647.165595896:main thread: Hdlr : 0x0 7647.165597852:main thread: Owner: 0x0 7647.165599807:main thread: 7647.165602042:main thread: Command 'actionresumeinterval': 7647.165603998:main thread: type : 6 7647.165605953:main thread: pData: 0x0 7647.165607909:main thread: Hdlr : 0x804ff40 7647.165609864:main thread: Owner: 0x0 7647.165611820:main thread: 7647.165614055:main thread: Command 'controlcharacterescapeprefix': 7647.165617966:main thread: type : 8 7647.165619922:main thread: pData: 0x8081338 7647.165621877:main thread: Hdlr : 0x0 7647.165623833:main thread: Owner: 0x0 7647.165625788:main thread: 7647.165628023:main thread: Command 'escapecontrolcharactersonreceive': 7647.165629979:main thread: type : 4 7647.165631934:main thread: pData: 0x808133c 7647.165633890:main thread: Hdlr : 0x0 7647.165635845:main thread: Owner: 0x0 7647.165637801:main thread: 7647.165640036:main thread: Command 'dropmsgswithmaliciousdnsptrrecords': 7647.165643947:main thread: type : 4 7647.165645902:main thread: pData: 0x8083c90 7647.165647858:main thread: Hdlr : 0x0 7647.165649814:main thread: Owner: 0x0 7647.165652049:main thread: 7647.168092024:main thread: Command 'droptrailinglfonreception': 7647.168093979:main thread: type : 4 7647.168095935:main thread: pData: 0x8081340 7647.168097890:main thread: Hdlr : 0x0 7647.168100125:main thread: Owner: 0x0 7647.168104036:main thread: 7647.168105992:main thread: Command 'template': 7647.168107947:main thread: type : 1 7647.168109903:main thread: pData: 0x0 7647.168112138:main thread: Hdlr : 0x806d1c0 7647.168114093:main thread: Owner: 0x0 7647.168116049:main thread: 7647.168118004:main thread: Command 'outchannel': 7647.168119960:main thread: type : 1 7647.168121916:main thread: pData: 0x1 7647.168123871:main thread: Hdlr : 0x806d1c0 7647.168126106:main thread: Owner: 0x0 7647.168130017:main thread: 7647.168131973:main thread: Command 'allowedsender': 7647.168133928:main thread: type : 1 7647.168135884:main thread: pData: 0x2 7647.168138119:main thread: Hdlr : 0x806d1c0 7647.168140074:main thread: Owner: 0x0 7647.168142030:main thread: 7647.168143986:main thread: Command 'modload': 7647.168145941:main thread: type : 1 7647.168147897:main thread: pData: 0x0 7647.168150132:main thread: Hdlr : 0x806d2a0 7647.168152087:main thread: Owner: 0x0 7647.168155998:main thread: 7647.168157954:main thread: Command 'includeconfig': 7647.168159909:main thread: type : 1 7647.168162144:main thread: pData: 0x0 7647.168164100:main thread: Hdlr : 0x806d350 7647.168166055:main thread: Owner: 0x0 7647.168168011:main thread: 7647.168169966:main thread: Command 'umask': 7647.168171922:main thread: type : 5 7647.168173877:main thread: pData: 0x0 7647.168176112:main thread: Hdlr : 0x804ff10 7647.168178068:main thread: Owner: 0x0 7647.168180024:main thread: 7647.168183935:main thread: Command 'debugprinttemplatelist': 7647.168185890:main thread: type : 4 7647.168188125:main thread: pData: 0x8081344 7647.168190081:main thread: Hdlr : 0x0 7647.168192036:main thread: Owner: 0x0 7647.168193992:main thread: 7647.174191961:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 7647.174200062:main thread: Command 'debugprintmodulelist': 7647.174202018:main thread: type : 4 7647.174203973:main thread: pData: 0x8081348 7647.174205929:main thread: Hdlr : 0x0 7647.174207885:main thread: Owner: 0x0 7647.174209840:main thread: 7647.174212075:main thread: Command 'debugprintcfsyslinehandlerlist': 7647.174214031:main thread: type : 4 7647.174215986:main thread: pData: 0x808134c 7647.174217942:main thread: Hdlr : 0x0 7647.174219897:main thread: Owner: 0x0 7647.175595212:main thread: 7647.175597167:main thread: Command 'moddir': 7647.175599123:main thread: type : 11 7647.175601078:main thread: pData: 0x8084da0 7647.175603313:main thread: Hdlr : 0x0 7647.175605269:main thread: Owner: 0x0 7647.175607224:main thread: 7647.175609180:main thread: Command 'actionqueuefilename': 7647.175611136:main thread: type : 11 7647.175613091:main thread: pData: 0x8084dd4 7647.175615326:main thread: Hdlr : 0x0 7647.175619237:main thread: Owner: 0x0 7647.175621193:main thread: 7647.175623148:main thread: Command 'actionqueuesize': 7647.175625104:main thread: type : 6 7647.175627339:main thread: pData: 0x8082c60 7647.175629294:main thread: Hdlr : 0x0 7647.175631250:main thread: Owner: 0x0 7647.175633206:main thread: 7647.175635161:main thread: Command 'actionqueuemaxdiskspace': 7647.175637117:main thread: type : 7 7647.175639072:main thread: pData: 0x8084dd8 7647.175641307:main thread: Hdlr : 0x0 7647.175643263:main thread: Owner: 0x0 7647.175647174:main thread: 7647.175649129:main thread: Command 'actionqueuehighwatermark': 7647.175651085:main thread: type : 6 7647.175653320:main thread: pData: 0x8082c64 7647.175655275:main thread: Hdlr : 0x0 7647.175657231:main thread: Owner: 0x0 7647.175659186:main thread: 7647.175661142:main thread: Command 'actionqueuelowwatermark': 7647.175663097:main thread: type : 6 7647.175665332:main thread: pData: 0x8082c68 7647.178391936:main thread: Hdlr : 0x0 7647.178393892:main thread: Owner: 0x0 7647.178396126:main thread: 7647.178398082:main thread: Command 'actionqueuediscardmark': 7647.178400038:main thread: type : 6 7647.178401993:main thread: pData: 0x8082c6c 7647.178403949:main thread: Hdlr : 0x0 7647.178405904:main thread: Owner: 0x0 7647.178408139:main thread: 7647.178410095:main thread: Command 'actionqueuediscardseverity': 7647.178412050:main thread: type : 6 7647.178414006:main thread: pData: 0x8082c70 7647.178417917:main thread: Hdlr : 0x0 7647.178420152:main thread: Owner: 0x0 7647.178422107:main thread: 7647.178424063:main thread: Command 'actionqueuecheckpointinterval': 7647.178426018:main thread: type : 6 7647.178427974:main thread: pData: 0x8084de0 7647.178429930:main thread: Hdlr : 0x0 7647.178432164:main thread: Owner: 0x0 7647.178434120:main thread: 7647.178436076:main thread: Command 'actionqueuetype': 7647.178438031:main thread: type : 11 7647.178439987:main thread: pData: 0x0 7647.178441942:main thread: Hdlr : 0x8074b50 7647.178443898:main thread: Owner: 0x0 7647.178446133:main thread: 7647.178448088:main thread: Command 'actionqueueworkerthreads': 7647.178450044:main thread: type : 6 7647.178452000:main thread: pData: 0x8082c74 7647.178453955:main thread: Hdlr : 0x0 7647.178455911:main thread: Owner: 0x0 7647.178458146:main thread: 7647.178460101:main thread: Command 'actionqueuetimeoutshutdown': 7647.178462057:main thread: type : 6 7647.178464012:main thread: pData: 0x8084de4 7647.178465968:main thread: Hdlr : 0x0 7647.178467923:main thread: Owner: 0x0 7647.178470158:main thread: 7647.178472114:main thread: Command 'actionqueuetimeoutactioncompletion': 7647.178474069:main thread: type : 6 7647.178476025:main thread: pData: 0x8082c78 7647.178477980:main thread: Hdlr : 0x0 7647.178479936:main thread: Owner: 0x0 7647.178481891:main thread: 7647.178484126:main thread: Command 'actionqueuetimeoutenqueue': 7647.179954704:main thread: type : 6 7647.179956939:main thread: pData: 0x8082c7c 7647.179958895:main thread: Hdlr : 0x0 7647.179960850:main thread: Owner: 0x0 7647.179962806:main thread: 7647.179964761:main thread: Command 'actionqueueworkertimeoutthreadshutdown': 7647.179966717:main thread: type : 6 7647.179968952:main thread: pData: 0x8082c80 7647.179970907:main thread: Hdlr : 0x0 7647.179972863:main thread: Owner: 0x0 7647.179974819:main thread: 7647.179976774:main thread: Command 'actionqueueworkerthreadminimummessages': 7647.179978730:main thread: type : 6 7647.179982920:main thread: pData: 0x8082c84 7647.179984876:main thread: Hdlr : 0x0 7647.179986831:main thread: Owner: 0x0 7647.179988787:main thread: 7647.179990743:main thread: Command 'actionqueuemaxfilesize': 7647.179992698:main thread: type : 7 7647.179994933:main thread: pData: 0x8082c88 7647.179996889:main thread: Hdlr : 0x0 7647.179998844:main thread: Owner: 0x0 7647.180000800:main thread: 7647.180002755:main thread: Command 'actionqueuesaveonshutdown': 7647.180004711:main thread: type : 4 7647.180008901:main thread: pData: 0x8082c90 7647.180010857:main thread: Hdlr : 0x0 7647.180012812:main thread: Owner: 0x0 7647.180014768:main thread: 7647.180016723:main thread: Command 'actionqueuedequeueslowdown': 7647.180018958:main thread: type : 6 7647.180020914:main thread: pData: 0x8084de8 7647.180022869:main thread: Hdlr : 0x0 7647.180024825:main thread: Owner: 0x0 7647.180028736:main thread: 7647.180030971:main thread: Command 'actionqueuedequeuetimebegin': 7647.180032927:main thread: type : 6 7647.180034882:main thread: pData: 0x8084dec 7647.180036838:main thread: Hdlr : 0x0 7647.180038793:main thread: Owner: 0x0 7647.180040749:main thread: 7647.180042704:main thread: Command 'actionqueuedequeuetimeend': 7647.180044939:main thread: type : 6 7647.180048850:main thread: pData: 0x8082c94 7647.180050806:main thread: Hdlr : 0x0 7647.180052762:main thread: Owner: 0x0 7647.185069600:main thread: 7647.185071835:main thread: 7647.185073791:main thread: Messages with malicious PTR DNS Records are not dropped. 7647.185075746:main thread: Control characters are replaced upon reception. 7647.185077702:main thread: Control character escape sequence prefix is '#'. 7647.185079657:main thread: Main queue size 10000 messages. 7647.185083848:main thread: Main queue worker threads: 1, Perists every 0 updates. 7647.185085803:main thread: Main queue timeouts: shutdown: 0, action completion shutdown: 1000, enq: 2000 7647.185087759:main thread: Main queue watermarks: high: 8000, low: 2000, discard: 9800, discard-severity: 4 7647.185089715:main thread: Main queue save on shutdown 1, max disk space allowed 0 7647.185091670:main thread: Work Directory: '(null)'. 7647.186613651:main thread: logmsg: flags 5, from 'qa-freebsd', msg [origin software="rsyslogd" swVersion="3.17.4" x-pid="17462" x-info="http://www.rsyslog.com"] restart 7647.186615607:main thread: Message has legacy syslog format. 7647.187085220:main thread: Signal 11 (SIGSEGV) occured, execution must be terminated. 7647.187193054:main thread: 7647.187196966:main thread: Recorded Call Order for Thread 'main thread (28201100)': 7647.187198921:main thread: maximum number of nested calls for this thread: 0. 7647.187200877:main thread: NOTE: not all calls may have been recorded, code does not currently guarantee that! 7647.187202832:main thread: 7647.187205067:main thread: Recorded Call Order for Thread 'action 1 queue:Reg/w0 (28201200)': 7647.187207023:main thread: maximum number of nested calls for this thread: 0. 7647.187208978:main thread: NOTE: not all calls may have been recorded, code does not currently guarantee that! 7647.187210934:main thread: 7647.187212890:main thread: Recorded Call Order for Thread 'main queue:Reg/w0 (28201300)': 7647.187214845:main thread: maximum number of nested calls for this thread: 0. 7647.187217080:main thread: NOTE: not all calls may have been recorded, code does not currently guarantee that! 7647.189175709:main thread: Mutex log for all known mutex operations: 7647.189177664:main thread: If the call trace is empty, you may want to ./configure --enable-rtinst 7647.189179620:main thread: To submit bug reports, visit http://www.rsyslog.com/bugs 7647.189602858:main thread: To submit bug reports, visit http://www.rsyslog.com/bugs Abort (core dumped)

Please let me know if I can provide further information.
-HKS

Learn more about rsyslog professional services.

Custom written rsyslog.conf?
Maintenance Contract?
Installation support?

by **rgerhards** » Wed Jul 02, 2008 7:05 am

Hi,

I tried to reproduce, but I can not

Also, the debug log does not contain sufficient information. Would it be possible that you get me a gdb backtrace of a crashed instance? That would be most helpful. As an alternative, please add --enable-rtinst to the configure command. That provides at least a bit more information on the rsyslog crashdump itself.

Thanks,
Rainer

by **hkspvt** » Wed Jul 02, 2008 4:40 pm

Here's the gdb backtrace. I'm not super-familiar with the tool, so let me know if you need further details, the full core file, etc.

Code: Select all: qa-freebsd# gdb /usr/local/sbin/rsyslogd rsyslogd.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Core was generated by `rsyslogd'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libz.so.4...done. Loaded symbols for /lib/libz.so.4 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmnet.so Reading symbols from /usr/local/lib/rsyslog/lmtcpclt.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmtcpclt.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x2811e5cb in thr_kill () from /lib/libc.so.7 [New Thread 0x28201300 (LWP 100091)] [New Thread 0x28201200 (LWP 100090)] [New Thread 0x28201100 (LWP 100082)] (gdb) bt #0 0x2811e5cb in thr_kill () from /lib/libc.so.7 #1 0x280d2386 in pthread_kill () from /lib/libthr.so.3 #2 0x280d0123 in raise () from /lib/libthr.so.3 #3 0x281abb8a in abort () from /lib/libc.so.7 #4 0x080581ec in sigsegvHdlr (signum=6) at debug.c:741 #5 <signal handler called> #6 0x280d156b in pthread_mutex_getprioceiling () from /lib/libthr.so.3 #7 0x0806a295 in MsgPrepareEnqueueLockingCase (pThis=0x0) at msg.c:167 #8 0x08050d25 in logmsg (pMsg=0x282244a0, flags=1) at syslogd.c:1629 #9 0x0805140f in logmsgInternal (pri=Variable "pri" is not available. ) at syslogd.c:915 #10 0x08051c57 in init () at syslogd.c:2370 #11 0x08053b16 in realMain (argc=5, argv=0xbfbfeccc) at syslogd.c:2808 #12 0x0805424f in main (argc=Cannot access memory at address 0x0 ) at syslogd.c:3483 (gdb) quit qa-freebsd#

Thanks for the help.
-HKS

by **rgerhards** » Wed Jul 09, 2008 10:16 am

hmmm... could you run configure with --enable-debug and --enable-rtinst and let me know the output of the run. I think it still aborts. The gdb backtrace did not point me as good as I hoped. We may need to look into the other threads, but the --enable-rtinst can probably aid in seeing what's going on.

Rainer

by **hkspvt** » Wed Jul 09, 2008 5:17 pm

The -dn output with --enable-debug and --enable-rtinst:

Code: Select all: 0403.071359216:28201100: Writing pidfile /var/run/rsyslogd.pid. 0403.073852829:28201100: rsyslog 3.17.4 - called init() 0403.073854785:28201100: Unloading non-static modules. 0403.073856740:28201100: module lmnet NOT unloaded because it still has a refcount of 3 0403.073858696:28201100: module lmtcpclt NOT unloaded because it still has a refcount of 1 0403.073860651:28201100: modules still loaded after module.UnloadAndDestructAll: 0403.073862607:28201100: printing users of loadable module lmnet, refcount 3, ptr 0x28216080, type 2 0403.073866797:28201100: module lmnet is currently in use by file omfwd.c 0403.073868753:28201100: module lmnet is currently in use by file syslogd.c 0403.073870708:28201100: module lmnet is currently in use by file conf.c 0403.073872664:28201100: printing users of loadable module builtin-file, refcount 0, ptr 0x282160e0, type 1 0403.073874620:28201100: printing users of loadable module lmtcpclt, refcount 1, ptr 0x282161a0, type 2 0403.073876854:28201100: module lmtcpclt is currently in use by file omfwd.c 0403.073878810:28201100: printing users of loadable module builtin-fwd, refcount 0, ptr 0x28216140, type 1 0403.073880766:28201100: printing users of loadable module builtin-shell, refcount 0, ptr 0x28216200, type 1 0403.073882721:28201100: printing users of loadable module builtin-discard, refcount 0, ptr 0x28216260, type 1 0403.073884677:28201100: printing users of loadable module builtin-usrmsg, refcount 0, ptr 0x282162c0, type 1 0403.073886632:28201100: Clearing templates. 0403.075387382:28201100: cfline: '$ActionQueueType LinkedList' 0403.075427331:28201100: action queue type set to LINKEDLIST 0403.075429287:28201100: cfline: '*.* /var/log/debug.log' 0403.075855598:28201100: - traditional PRI filter 0403.075857553:28201100: symbolic name: * ==> 255 0403.076348677:28201100: tried selector action for builtin-file: 0 0403.076350633:28201100: Module builtin-file processed this config line. 0403.076352588:28201100: template: 'RSYSLOG_FileFormat' assigned 0403.076870810:28201100: action 1 queue: save on shutdown 1, max disk space allowed 0 0403.076930315:28201100: action 1 queue: is NOT disk-assisted 0403.076934226:28201100: action 1 queue: type 1, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 0403.077326455:28201100: action 1 queue:Reg: finalizing construction of worker thread pool 0403.077328410:28201100: action 1 queue:Reg/w0: finalizing construction of worker instance data 0403.077330645:28201100: action 1 queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 0403.077416411:28201100: action 1 queue:Reg: high activity - starting 1 additional worker thread(s). 0403.077796627:28201100: action 1 queue:Reg/w0: receiving command 2 0403.078289706:28201100: action 1 queue:Reg: started with state 0, num workers now 1 0403.080407014:28201100: Action 0x282240b0: queue 0x28229260 created 0403.080884449:28201100: selector line successfully processed 0403.080948144:28201100: main queue: is NOT disk-assisted 0403.080952335:28201100: main queue: type 0, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 0403.081337859:28201100: main queue:Reg: finalizing construction of worker thread pool 0403.081339814:28201100: main queue:Reg/w0: finalizing construction of worker instance data 0403.081341770:28201100: main queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 0403.081343725:28201100: main queue:Reg: high activity - starting 1 additional worker thread(s). 0403.081345681:28201100: main queue:Reg/w0: receiving command 2 0403.081896868:28201100: main queue:Reg: started with state 0, num workers now 1 0403.081947433:28201200: action 1 queue:Reg/w0: receiving command 4 0403.083365770:28201100: Main processing queue is initialized and running 0403.083398735:28201100: Active selectors: 0403.083437008:28201100: Selector 1: 0403.083438964:28201100: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Actions: 0403.084835510:28201100: builtin-file: /var/log/debug.log Instance data: 0x28229140 0403.084891104:28201100: RepeatedMsgReduction: 0 0403.084893059:28201100: Resume Interval: 30 0403.084895294:28201100: Suspended: 0 0403.085335573:28201100: Disabled: 0 0403.085337529:28201100: Exec only when previous is suspended: 0 0403.085339484:28201100: 0403.085341719:28201100: 0403.085343675:28201100: 0403.085345630:28201100: 0403.085347586:28201100: Template: Name='RSYSLOG_SyslogProtocol23Format' 0403.085866367:28201100: Entry(28205700): type 1, (CONSTANT), value: '<' 0403.085919726:28201100: Entry(28205740): type 2, (FIELD), value: 'pri' 0403.085923637:28201100: Entry(28205780): type 1, (CONSTANT), value: '>1 ' 0403.086374812:28201100: Entry(282057c0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 0403.086442139:28201100: Entry(28205800): type 1, (CONSTANT), value: ' ' 0403.086824031:28201100: Entry(28205840): type 2, (FIELD), value: 'hostname' 0403.086877390:28201100: Entry(28205880): type 1, (CONSTANT), value: ' ' 0403.086931028:28201100: Entry(282058c0): type 2, (FIELD), value: 'app-name' 0403.086934939:28201100: Entry(28205900): type 1, (CONSTANT), value: ' ' 0403.087374101:28201100: Entry(28205940): type 2, (FIELD), value: 'procid' 0403.087428018:28201100: Entry(28205980): type 1, (CONSTANT), value: ' ' 0403.087809910:28201100: Entry(282059c0): type 2, (FIELD), value: 'msgid' 0403.087863269:28201100: Entry(28205a00): type 1, (CONSTANT), value: ' ' 0403.087916348:28201100: Entry(28205a40): type 2, (FIELD), value: 'structured-data' 0403.088290977:28201100: Entry(28205a80): type 1, (CONSTANT), value: ' ' 0403.088359422:28201100: Entry(28205ac0): type 2, (FIELD), value: 'msg' 0403.088412780:28201100: Entry(28205b00): type 1, (CONSTANT), value: ' ' 0403.089289987:28201100: Template: Name='RSYSLOG_FileFormat' 0403.089345860:28201100: Entry(28205b40): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 0403.089412628:28201100: Entry(28205b80): type 1, (CONSTANT), value: ' ' 0403.089796196:28201100: Entry(28205bc0): type 2, (FIELD), value: 'hostname' 0403.089849834:28201100: Entry(28205c00): type 1, (CONSTANT), value: ' ' 0403.089903193:28201100: Entry(28205c40): type 2, (FIELD), value: 'syslogtag' 0403.090290393:28201100: Entry(28205c80): type 2, (FIELD), value: 'msg' [drop last LF in msg] 0403.090359396:28201100: Entry(28205cc0): type 1, (CONSTANT), value: ' ' 0403.090424209:28201100: Template: Name='RSYSLOG_TraditionalFileFormat' 0403.090793809:28201100: Entry(28205d00): type 2, (FIELD), value: 'timestamp' 0403.090847168:28201100: Entry(28205d40): type 1, (CONSTANT), value: ' ' 0403.090900527:28201100: Entry(28205d80): type 2, (FIELD), value: 'hostname' 0403.091290241:28201100: Entry(28205dc0): type 1, (CONSTANT), value: ' ' 0403.091345555:28201100: Entry(28205e00): type 2, (FIELD), value: 'syslogtag' 0403.091398914:28201100: Entry(28205e40): type 2, (FIELD), value: 'msg' [drop last LF in msg] 0403.091795333:28201100: Entry(28205e80): type 1, (CONSTANT), value: ' ' 0403.091870203:28201100: Template: Name=' WallFmt' 0403.091909593:28201100: Entry(28205ec0): type 1, (CONSTANT), value: ' Message from syslogd@' 0403.092331993:28201100: Entry(28205f00): type 2, (FIELD), value: 'hostname' 0403.092385911:28201100: Entry(28205f40): type 1, (CONSTANT), value: ' at ' 0403.092438990:28201100: Entry(28205f80): type 2, (FIELD), value: 'timegenerated' 0403.092826190:28201100: Entry(28205fc0): type 1, (CONSTANT), value: ' ... ' 0403.092890724:28201100: Entry(28222040): type 2, (FIELD), value: 'syslogtag' 0403.092944082:28201100: Entry(28222080): type 2, (FIELD), value: 'msg' 0403.093332120:28201100: Entry(282220c0): type 1, (CONSTANT), value: ' ' 0403.093884146:28201100: Template: Name='RSYSLOG_ForwardFormat' 0403.093925492:28201100: Entry(28222100): type 1, (CONSTANT), value: '<' 0403.094288946:28201100: Entry(28222140): type 2, (FIELD), value: 'pri' 0403.094344540:28201100: Entry(28222180): type 1, (CONSTANT), value: '>' 0403.094398178:28201100: Entry(282221c0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 0403.094817226:28201100: Entry(28222200): type 1, (CONSTANT), value: ' ' 0403.094870863:28201100: Entry(28222240): type 2, (FIELD), value: 'hostname' 0403.094924222:28201100: Entry(28222280): type 1, (CONSTANT), value: ' ' 0403.095329302:28201100: Entry(282222c0): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 0403.095396908:28201100: Entry(28222300): type 2, (FIELD), value: 'msg' 0403.095781035:28201100: Template: Name='RSYSLOG_TraditionalForwardFormat' 0403.095821264:28201100: Entry(28222340): type 1, (CONSTANT), value: '<' 0403.095874622:28201100: Entry(28222380): type 2, (FIELD), value: 'pri' 0403.095927981:28201100: Entry(282223c0): type 1, (CONSTANT), value: '>' 0403.096340883:28201100: Entry(28222400): type 2, (FIELD), value: 'timestamp' 0403.096394521:28201100: Entry(28222440): type 1, (CONSTANT), value: ' ' 0403.096447880:28201100: Entry(28222480): type 2, (FIELD), value: 'hostname' 0403.096830889:28201100: Entry(282224c0): type 1, (CONSTANT), value: ' ' 0403.096884248:28201100: Entry(28222500): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 0403.097287651:28201100: Entry(28222540): type 2, (FIELD), value: 'msg' 0403.097343245:28201100: Template: Name=' StdUsrMsgFmt' 0403.097382635:28201100: Entry(28222580): type 1, (CONSTANT), value: ' ' 0403.097780731:28201100: Entry(282225c0): type 2, (FIELD), value: 'syslogtag' 0403.097834927:28201100: Entry(28222600): type 2, (FIELD), value: 'msg' 0403.097888007:28201100: Entry(28222640): type 1, (CONSTANT), value: ' ' 0403.098289455:28201100: Template: Name=' StdDBFmt' [SQL-Format (MySQL)] 0403.099401328:28201100: Entry(28222680): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 0403.099458877:28201100: Entry(282226c0): type 2, (FIELD), value: 'msg' 0403.099829874:28201100: Entry(28222700): type 1, (CONSTANT), value: '', ' 0403.099883232:28201100: Entry(28222740): type 2, (FIELD), value: 'syslogfacility' 0403.099936312:28201100: Entry(28222780): type 1, (CONSTANT), value: ', '' 0403.099940502:28201100: Entry(282227c0): type 2, (FIELD), value: 'hostname' 0403.100404528:28201100: Entry(28222800): type 1, (CONSTANT), value: '', ' 0403.100784464:28201100: Entry(28222840): type 2, (FIELD), value: 'syslogpriority' 0403.100837823:28201100: Entry(28222880): type 1, (CONSTANT), value: ', '' 0403.100890903:28201100: Entry(282228c0): type 2, (FIELD), value: 'timereported' [Format as MySQL-Date] 0403.101291233:28201100: Entry(28222900): type 1, (CONSTANT), value: '', '' 0403.101383703:28201100: Entry(28222940): type 2, (FIELD), value: 'timegenerated' [Format as MySQL-Date] 0403.101450750:28201100: Entry(28222980): type 1, (CONSTANT), value: '', ' 0403.101833201:28201100: Entry(282229c0): type 2, (FIELD), value: 'iut' 0403.101886280:28201100: Entry(28222a00): type 1, (CONSTANT), value: ', '' 0403.101939360:28201100: Entry(28222a40): type 2, (FIELD), value: 'syslogtag' 0403.102329633:28201100: Entry(28222a80): type 1, (CONSTANT), value: '')' 0403.102382991:28201100: Template: Name=' StdPgSQLFmt' [SQL-Format (standard SQL)] 0403.102435233:28201100: Entry(28222ac0): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 0403.102821315:28201100: Entry(28222b00): type 2, (FIELD), value: 'msg' 0403.102874674:28201100: Entry(28222b40): type 1, (CONSTANT), value: '', ' 0403.102927753:28201100: Entry(28222b80): type 2, (FIELD), value: 'syslogfacility' 0403.103332553:28201100: Entry(28222bc0): type 1, (CONSTANT), value: ', '' 0403.104386039:28201100: Entry(28222c00): type 2, (FIELD), value: 'hostname' 0403.104440516:28201100: Entry(28222c40): type 1, (CONSTANT), value: '', ' 0403.104825201:28201100: Entry(28222c80): type 2, (FIELD), value: 'syslogpriority' 0403.104878281:28201100: Entry(28222cc0): type 1, (CONSTANT), value: ', '' 0403.104931360:28201100: Entry(28222d00): type 2, (FIELD), value: 'timereported' [Format as PgSQL-Date] 0403.105346776:28201100: Entry(28222d40): type 1, (CONSTANT), value: '', '' 0403.105400135:28201100: Entry(28222d80): type 2, (FIELD), value: 'timegenerated' [Format as PgSQL-Date] 0403.106086255:28201100: Entry(28222dc0): type 1, (CONSTANT), value: '', ' 0403.106142687:28201100: Entry(28222e00): type 2, (FIELD), value: 'iut' 0403.106196605:28201100: Entry(28222e40): type 1, (CONSTANT), value: ', '' 0403.106249684:28201100: Entry(28222e80): type 2, (FIELD), value: 'syslogtag' 0403.106253874:28201100: Entry(28222ec0): type 1, (CONSTANT), value: '')' 0403.106633532:28201100: Loaded Module: Name='lmnet', IFVersion=4, type=library module. 0403.106818751:28201100: Entry points: 0403.106820706:28201100: queryEtryPt: 0x281ee6b0 0403.106822662:28201100: doAction: 0x0 0403.106824618:28201100: parseSelectorAct: 0x0 0403.106826573:28201100: dbgPrintInstInfo: 0x0 0403.106828808:28201100: freeInstance: 0x0 0403.106832719:28201100: 0403.106834675:28201100: Loaded Module: Name='builtin-file', IFVersion=4, type=output module. 0403.107399551:28201100: Entry points: 0403.107401506:28201100: queryEtryPt: 0x8081fd0 0403.107403462:28201100: doAction: 0x8083280 0403.107405418:28201100: parseSelectorAct: 0x8082780 0403.107407373:28201100: dbgPrintInstInfo: 0x8083020 0403.107409608:28201100: freeInstance: 0x8082630 0403.107411564:28201100: 0403.107413519:28201100: Loaded Module: Name='lmtcpclt', IFVersion=4, type=library module. 0403.108334865:28201100: Entry points: 0403.108336821:28201100: queryEtryPt: 0x281f3290 0403.109469926:28201100: doAction: 0x0 0403.109471881:28201100: parseSelectorAct: 0x0 0403.109473837:28201100: dbgPrintInstInfo: 0x0 0403.109475792:28201100: freeInstance: 0x0 0403.109478027:28201100: 0403.109479983:28201100: Loaded Module: Name='builtin-fwd', IFVersion=4, type=output module. 0403.109947361:28201100: Entry points: 0403.109951551:28201100: queryEtryPt: 0x8080840 0403.109953507:28201100: doAction: 0x8080fc0 0403.109955462:28201100: parseSelectorAct: 0x8081400 0403.109957418:28201100: dbgPrintInstInfo: 0x8080f50 0403.109959373:28201100: freeInstance: 0x8080e70 0403.109961329:28201100: 0403.109963564:28201100: Loaded Module: Name='builtin-shell', IFVersion=4, type=output module. 0403.110875412:28201100: Entry points: 0403.110877367:28201100: queryEtryPt: 0x807f1e0 0403.110879323:28201100: doAction: 0x807f6d0 0403.110883513:28201100: parseSelectorAct: 0x807f410 0403.110885469:28201100: dbgPrintInstInfo: 0x807f660 0403.110887424:28201100: freeInstance: 0x807f3a0 0403.110889380:28201100: 0403.110891335:28201100: Loaded Module: Name='builtin-discard', IFVersion=4, type=output module. 0403.111793685:28201100: Entry points: 0403.111795640:28201100: queryEtryPt: 0x8083bc0 0403.111797875:28201100: doAction: 0x8083d80 0403.111799831:28201100: parseSelectorAct: 0x8083e50 0403.111801786:28201100: dbgPrintInstInfo: 0x8083b70 0403.111803742:28201100: freeInstance: 0x8083de0 0403.111807653:28201100: 0403.111809888:28201100: Loaded Module: Name='builtin-usrmsg', IFVersion=4, type=output module. 0403.112396555:28201100: Entry points: 0403.112398790:28201100: queryEtryPt: 0x807fa20 0403.112400745:28201100: doAction: 0x8080410 0403.112402701:28201100: parseSelectorAct: 0x807fce0 0403.112404656:28201100: dbgPrintInstInfo: 0x807fbe0 0403.112406612:28201100: freeInstance: 0x807fc70 0403.112408567:28201100: 0403.113890320:28201100: Sytem Line Configuration Commands: 0403.113892276:28201100: Command 'dynafilecachesize': 0403.113894231:28201100: type : 6 0403.113898142:28201100: pData: 0x0 0403.113900377:28201100: Hdlr : 0x8082190 0403.113902333:28201100: Owner: 0x8082420 0403.113904288:28201100: 0403.113906244:28201100: Command 'dirowner': 0403.113908199:28201100: type : 2 0403.113910155:28201100: pData: 0x809c588 0403.113912390:28201100: Hdlr : 0x0 0403.113914345:28201100: Owner: 0x8082420 0403.113916301:28201100: 0403.113918256:28201100: Command 'dirgroup': 0403.113920212:28201100: type : 3 0403.113922168:28201100: pData: 0x809c58c 0403.115312009:28201100: Hdlr : 0x0 0403.115313965:28201100: Owner: 0x8082420 0403.115315920:28201100: 0403.115317876:28201100: Command 'fileowner': 0403.115795311:28201100: type : 2 0403.115797266:28201100: pData: 0x809c580 0403.115799501:28201100: Hdlr : 0x0 0403.115801457:28201100: Owner: 0x8082420 0403.115803412:28201100: 0403.115805368:28201100: Command 'filegroup': 0403.115809279:28201100: type : 3 0403.115811235:28201100: pData: 0x809c584 0403.115813469:28201100: Hdlr : 0x0 0403.115815425:28201100: Owner: 0x8082420 0403.115817381:28201100: 0403.115819336:28201100: Command 'dircreatemode': 0403.115821292:28201100: type : 5 0403.115823247:28201100: pData: 0x8099bd0 0403.115825482:28201100: Hdlr : 0x0 0403.115827438:28201100: Owner: 0x8082420 0403.115829393:28201100: 0403.115831349:28201100: Command 'filecreatemode': 0403.115835260:28201100: type : 5 0403.115837495:28201100: pData: 0x8099bd4 0403.115839450:28201100: Hdlr : 0x0 0403.115841406:28201100: Owner: 0x8082420 0403.115843361:28201100: 0403.115845317:28201100: Command 'createdirs': 0403.115847273:28201100: type : 4 0403.115849228:28201100: pData: 0x809c590 0403.115851463:28201100: Hdlr : 0x0 0403.115853419:28201100: Owner: 0x8082420 0403.115857330:28201100: 0403.115859285:28201100: Command 'failonchownfailure': 0403.119809229:28201100: type : 4 0403.119811184:28201100: pData: 0x809c57c 0403.119813140:28201100: Hdlr : 0x0 0403.119815095:28201100: Owner: 0x8082420 0403.119817051:28201100: 0403.119821241:28201100: Command 'actionfileenablesync': 0403.119823197:28201100: type : 4 0403.119825152:28201100: pData: 0x809c534 0403.119827108:28201100: Hdlr : 0x0 0403.119829064:28201100: Owner: 0x8082420 0403.119831019:28201100: 0403.119833254:28201100: Command 'actionfiledefaulttemplate': 0403.119835210:28201100: type : 11 0403.119837165:28201100: pData: 0x809c538 0403.119839121:28201100: Hdlr : 0x0 0403.119841076:28201100: Owner: 0x0 0403.119844987:28201100: 0403.119847222:28201100: Command 'resetconfigvariables': 0403.119849178:28201100: type : 1 0403.119851133:28201100: pData: 0x0 0403.119853089:28201100: Hdlr : 0x8082480 0403.119855045:28201100: Owner: 0x8082420 0403.119857000:28201100: 0403.119859235:28201100: type : 1 0403.119861191:28201100: pData: 0x0 0403.119863146:28201100: Hdlr : 0x8080d60 0403.119867057:28201100: Owner: 0x8080d90 0403.119869013:28201100: 0403.119871248:28201100: type : 1 0403.119873203:28201100: pData: 0x0 0403.119875159:28201100: Hdlr : 0x804faa0 0403.119877114:28201100: Owner: 0x0 0403.119879070:28201100: 0403.119881025:28201100: Command 'actionforwarddefaulttemplate': 0403.119882981:28201100: type : 11 0403.119885216:28201100: pData: 0x809c4f0 0403.119889127:28201100: Hdlr : 0x0 0403.119891083:28201100: Owner: 0x0 0403.119893038:28201100: 0403.119894994:28201100: Command 'workdirectory': 0403.119897229:28201100: type : 11 0403.119899184:28201100: pData: 0x809ae28 0403.119901140:28201100: Hdlr : 0x0 0403.119903095:28201100: Owner: 0x0 0403.119905051:28201100: 0403.119907006:28201100: Command 'actionresumeretrycount': 0403.119909241:28201100: type : 6 0403.119913152:28201100: pData: 0x809c6d0 0403.119915108:28201100: Hdlr : 0x0 0403.119917063:28201100: Owner: 0x0 0403.119919019:28201100: 0403.125829826:28201100: Command 'mainmsgqueuefilename': 0403.125831782:28201100: type : 11 0403.125834017:28201100: pData: 0x809ae60 0403.125835972:28201100: Hdlr : 0x0 0403.125837928:28201100: Owner: 0x0 0403.125841839:28201100: 0403.125843795:28201100: Command 'mainmsgqueuesize': 0403.125846029:28201100: type : 6 0403.125847985:28201100: pData: 0x8098480 0403.125849941:28201100: Hdlr : 0x0 0403.125851896:28201100: Owner: 0x0 0403.125853852:28201100: 0403.125855807:28201100: Command 'mainmsgqueuehighwatermark': 0403.125858042:28201100: type : 6 0403.125859998:28201100: pData: 0x8098484 0403.125861953:28201100: Hdlr : 0x0 0403.125863909:28201100: Owner: 0x0 0403.125867820:28201100: 0403.125869775:28201100: Command 'mainmsgqueuelowwatermark': 0403.125872010:28201100: type : 6 0403.125873966:28201100: pData: 0x8098488 0403.125875921:28201100: Hdlr : 0x0 0403.125877877:28201100: Owner: 0x0 0403.125879833:28201100: 0403.125881788:28201100: Command 'mainmsgqueuediscardmark': 0403.125884023:28201100: type : 6 0403.125885979:28201100: pData: 0x809848c 0403.125887934:28201100: Hdlr : 0x0 0403.125889890:28201100: Owner: 0x0 0403.125893801:28201100: 0403.125896036:28201100: Command 'mainmsgqueuediscardseverity': 0403.125897991:28201100: type : 10 0403.125899947:28201100: pData: 0x8098490 0403.125901903:28201100: Hdlr : 0x0 0403.125903858:28201100: Owner: 0x0 0403.125905814:28201100: 0403.125907769:28201100: Command 'mainmsgqueuecheckpointinterval': 0403.125910004:28201100: type : 6 0403.125911960:28201100: pData: 0x809ae64 0403.125913915:28201100: Hdlr : 0x0 0403.125915871:28201100: Owner: 0x0 0403.125917826:28201100: 0403.125922017:28201100: Command 'mainmsgqueuetype': 0403.125923972:28201100: type : 11 0403.125925928:28201100: pData: 0x0 0403.125927883:28201100: Hdlr : 0x8050170 0403.125929839:28201100: Owner: 0x0 0403.125931794:28201100: 0403.125934029:28201100: Command 'mainmsgqueueworkerthreads': 0403.088450774:action 1 queue:Reg/w0: action 1 queue: entering rate limiter 0403.133393078:28201300: main queue:Reg/w0: receiving command 4 0403.133458449:28201100: type : 6 0403.133460684:28201100: pData: 0x8098494 0403.133462640:28201100: Hdlr : 0x0 0403.133464596:28201100: Owner: 0x0 0403.133466551:28201100: 0403.133468507:28201100: Command 'mainmsgqueuetimeoutshutdown': 0403.133470462:28201100: type : 6 0403.133474653:28201100: pData: 0x809ae68 0403.133476608:28201100: Hdlr : 0x0 0403.133478564:28201100: Owner: 0x0 0403.133480519:28201100: 0403.133482475:28201100: Command 'mainmsgqueuetimeoutactioncompletion': 0403.133484430:28201100: type : 6 0403.133486665:28201100: pData: 0x8098498 0403.133488621:28201100: Hdlr : 0x0 0403.133490576:28201100: Owner: 0x0 0403.133492532:28201100: 0403.133494487:28201100: Command 'mainmsgqueuetimeoutenqueue': 0403.133496443:28201100: type : 6 0403.133498678:28201100: pData: 0x809849c 0403.135313434:28201100: Hdlr : 0x0 0403.135315669:28201100: Owner: 0x0 0403.135317624:28201100: 0403.135319580:28201100: Command 'mainmsgqueueworketimeoutrthreadshutdown': 0403.135321535:28201100: type : 6 0403.135323491:28201100: pData: 0x80984a0 0403.135325446:28201100: Hdlr : 0x0 0403.135327402:28201100: Owner: 0x0 0403.135329637:28201100: 0403.135331592:28201100: Command 'mainmsgqueuedequeueslowdown': 0403.135333548:28201100: type : 6 0403.136445142:28201100: pData: 0x809ae6c 0403.136447097:28201100: Hdlr : 0x0 0403.136449053:28201100: Owner: 0x0 0403.136451008:28201100: 0403.136453243:28201100: Command 'mainmsgqueueworkerthreadminimummessages': 0403.136455199:28201100: type : 6 0403.136457154:28201100: pData: 0x80984a4 0403.136459110:28201100: Hdlr : 0x0 0403.136461066:28201100: Owner: 0x0 0403.136463021:28201100: 0403.136465256:28201100: Command 'mainmsgqueuemaxfilesize': 0403.136467212:28201100: type : 7 0403.136469167:28201100: pData: 0x80984a8 0403.136471123:28201100: Hdlr : 0x0 0403.136473078:28201100: Owner: 0x0 0403.136475034:28201100: 0403.136476990:28201100: Command 'mainmsgqueuemaxdiskspace': 0403.136561358:28201100: type : 7 0403.136563313:28201100: pData: 0x809ae70 0403.136565269:28201100: Hdlr : 0x0 0403.136567224:28201100: Owner: 0x0 0403.136569180:28201100: 0403.136571415:28201100: Command 'mainmsgqueuesaveonshutdown': 0403.136573371:28201100: type : 4 0403.136575326:28201100: pData: 0x80984b0 0403.136577282:28201100: Hdlr : 0x0 0403.136579237:28201100: Owner: 0x0 0403.136581193:28201100: 0403.136585383:28201100: Command 'mainmsgqueuedequeuetimebegin': 0403.136587339:28201100: type : 6 0403.136589294:28201100: pData: 0x809ae78 0403.136591250:28201100: Hdlr : 0x0 0403.136593205:28201100: Owner: 0x0 0403.136595440:28201100: 0403.136597396:28201100: Command 'mainmsgqueuedequeuetimeend': 0403.136599351:28201100: type : 6 0403.136601307:28201100: pData: 0x80984b4 0403.136603262:28201100: Hdlr : 0x0 0403.136605218:28201100: Owner: 0x0 0403.136607174:28201100: 0403.136611364:28201100: Command 'repeatedmsgreduction': 0403.136613320:28201100: type : 4 0403.136615275:28201100: pData: 0x809c738 0403.136617231:28201100: Hdlr : 0x0 0403.136619186:28201100: Owner: 0x0 0403.136621421:28201100: 0403.136623377:28201100: Command 'actionexeconlywhenpreviousissuspended': 0403.136625332:28201100: type : 4 0403.136627288:28201100: pData: 0x809c73c 0403.136629244:28201100: Hdlr : 0x0 0403.136631199:28201100: Owner: 0x0 0403.136633434:28201100: 0403.136635390:28201100: Command 'actionexeconlyonceeveryinterval': 0403.136639301:28201100: type : 6 0403.136641256:28201100: pData: 0x809ae24 0403.136643212:28201100: Hdlr : 0x0 0403.136645447:28201100: Owner: 0x0 0403.136647402:28201100: 0403.136649358:28201100: Command 'actionresumeinterval': 0403.136651313:28201100: type : 6 0403.136653269:28201100: pData: 0x0 0403.136655224:28201100: Hdlr : 0x8050160 0403.136657180:28201100: Owner: 0x0 0403.136659415:28201100: 0403.136661370:28201100: Command 'controlcharacterescapeprefix': 0403.140315466:main queue:Reg/w0: main queue: entering rate limiter 0403.143407759:action 1 queue:Reg/w0: action 1 queue:Reg/w0: worker IDLE, waiting for work. 0403.143789651:28201100: type : 8 0403.143791606:28201100: pData: 0x80984b8 0403.143793841:28201100: Hdlr : 0x0 0403.143795797:28201100: Owner: 0x0 0403.143797752:28201100: 0403.143801664:28201100: Command 'escapecontrolcharactersonreceive': 0403.143803619:28201100: type : 4 0403.143805854:28201100: pData: 0x80984bc 0403.143807810:28201100: Hdlr : 0x0 0403.143809765:28201100: Owner: 0x0 0403.143811721:28201100: 0403.143813676:28201100: Command 'dropmsgswithmaliciousdnsptrrecords': 0403.143815632:28201100: type : 4 0403.143817867:28201100: pData: 0x809ae20 0403.143819822:28201100: Hdlr : 0x0 0403.143821778:28201100: Owner: 0x0 0403.143823733:28201100: 0403.145318895:28201100: Command 'droptrailinglfonreception': 0403.145320851:28201100: type : 4 0403.145322806:28201100: pData: 0x80984c0 0403.145324762:28201100: Hdlr : 0x0 0403.145326717:28201100: Owner: 0x0 0403.145328673:28201100: 0403.145330908:28201100: Command 'template': 0403.145332864:28201100: type : 1 0403.145334819:28201100: pData: 0x0 0403.145336775:28201100: Hdlr : 0x807e8c0 0403.145338730:28201100: Owner: 0x0 0403.145342921:28201100: 0403.145344876:28201100: Command 'outchannel': 0403.145346832:28201100: type : 1 0403.145348788:28201100: pData: 0x1 0403.145350743:28201100: Hdlr : 0x807e8c0 0403.145352699:28201100: Owner: 0x0 0403.145354654:28201100: 0403.145356889:28201100: Command 'allowedsender': 0403.145358845:28201100: type : 1 0403.145360800:28201100: pData: 0x2 0403.145362756:28201100: Hdlr : 0x807e8c0 0403.145364711:28201100: Owner: 0x0 0403.145366667:28201100: 0403.145370857:28201100: Command 'modload': 0403.145372813:28201100: type : 1 0403.145374768:28201100: pData: 0x0 0403.145376724:28201100: Hdlr : 0x807ea50 0403.145378679:28201100: Owner: 0x0 0403.145380914:28201100: 0403.145382870:28201100: Command 'includeconfig': 0403.145384825:28201100: type : 1 0403.147907213:28201100: pData: 0x0 0403.147909169:28201100: Hdlr : 0x807eba0 0403.147913080:28201100: Owner: 0x0 0403.147915035:28201100: 0403.147916991:28201100: Command 'umask': 0403.147919226:28201100: type : 5 0403.147921181:28201100: pData: 0x0 0403.147923137:28201100: Hdlr : 0x8050130 0403.147925093:28201100: Owner: 0x0 0403.147927048:28201100: 0403.147929004:28201100: Command 'debugprinttemplatelist': 0403.147931239:28201100: type : 4 0403.147933194:28201100: pData: 0x80984c4 0403.147935150:28201100: Hdlr : 0x0 0403.147937105:28201100: Owner: 0x0 0403.147941016:28201100: 0403.147943251:28201100: Command 'debugprintmodulelist': 0403.147945207:28201100: type : 4 0403.147947162:28201100: pData: 0x80984c8 0403.147949118:28201100: Hdlr : 0x0 0403.147951073:28201100: Owner: 0x0 0403.147953029:28201100: 0403.147955264:28201100: Command 'debugprintcfsyslinehandlerlist': 0403.147957219:28201100: type : 4 0403.147959175:28201100: pData: 0x80984cc 0403.147961131:28201100: Hdlr : 0x0 0403.147963086:28201100: Owner: 0x0 0403.147966997:28201100: 0403.147969232:28201100: Command 'moddir': 0403.147971188:28201100: type : 11 0403.147973143:28201100: pData: 0x809c5bc 0403.147975099:28201100: Hdlr : 0x0 0403.147977054:28201100: Owner: 0x0 0403.147979010:28201100: 0403.147981245:28201100: Command 'actionqueuefilename': 0403.147983200:28201100: type : 11 0403.147985156:28201100: pData: 0x809c6dc 0403.147987112:28201100: Hdlr : 0x0 0403.147989067:28201100: Owner: 0x0 0403.147991023:28201100: 0403.147995213:28201100: Command 'actionqueuesize': 0403.147997169:28201100: type : 6 0403.147999124:28201100: pData: 0x8099de0 0403.148001080:28201100: Hdlr : 0x0 0403.148003035:28201100: Owner: 0x0 0403.148004991:28201100: 0403.148007226:28201100: Command 'actionqueuemaxdiskspace': 0403.148009181:28201100: type : 7 0403.148011137:28201100: pData: 0x809c6e0 0403.148013093:28201100: Hdlr : 0x0 0403.148015048:28201100: Owner: 0x0 0403.154337081:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 0403.154343227:28201100: 0403.154345182:28201100: Command 'actionqueuehighwatermark': 0403.154347138:28201100: type : 6 0403.154349093:28201100: pData: 0x8099de4 0403.154351049:28201100: Hdlr : 0x0 0403.154353284:28201100: Owner: 0x0 0403.154355239:28201100: 0403.154357195:28201100: Command 'actionqueuelowwatermark': 0403.154361106:28201100: type : 6 0403.154363061:28201100: pData: 0x8099de8 0403.154365296:28201100: Hdlr : 0x0 0403.154367252:28201100: Owner: 0x0 0403.154369208:28201100: 0403.155835595:28201100: Command 'actionqueuediscardmark': 0403.155837830:28201100: type : 6 0403.155839786:28201100: pData: 0x8099dec 0403.155841741:28201100: Hdlr : 0x0 0403.155843697:28201100: Owner: 0x0 0403.156338173:28201100: 0403.156340408:28201100: Command 'actionqueuediscardseverity': 0403.156342363:28201100: type : 6 0403.156344319:28201100: pData: 0x8099df0 0403.156346275:28201100: Hdlr : 0x0 0403.156348230:28201100: Owner: 0x0 0403.156350186:28201100: 0403.156352141:28201100: Command 'actionqueuecheckpointinterval': 0403.156354376:28201100: type : 6 0403.156356332:28201100: pData: 0x809c6e8 0403.156360243:28201100: Hdlr : 0x0 0403.156362198:28201100: Owner: 0x0 0403.156364154:28201100: 0403.156366389:28201100: Command 'actionqueuetype': 0403.156368344:28201100: type : 11 0403.156370300:28201100: pData: 0x0 0403.156372256:28201100: Hdlr : 0x8088ae0 0403.156374211:28201100: Owner: 0x0 0403.156376167:28201100: 0403.156378402:28201100: Command 'actionqueueworkerthreads': 0403.156380357:28201100: type : 6 0403.156382313:28201100: pData: 0x8099df4 0403.156384268:28201100: Hdlr : 0x0 0403.156388179:28201100: Owner: 0x0 0403.156390414:28201100: 0403.156392370:28201100: Command 'actionqueuetimeoutshutdown': 0403.156394325:28201100: type : 6 0403.156396281:28201100: pData: 0x809c6ec 0403.156398236:28201100: Hdlr : 0x0 0403.156400192:28201100: Owner: 0x0 0403.158870059:28201100: 0403.158872015:28201100: Command 'actionqueuetimeoutactioncompletion': 0403.158873970:28201100: type : 6 0403.158877881:28201100: pData: 0x8099df8 0403.158879837:28201100: Hdlr : 0x0 0403.158882072:28201100: Owner: 0x0 0403.158884027:28201100: 0403.158885983:28201100: Command 'actionqueuetimeoutenqueue': 0403.158887938:28201100: type : 6 0403.158889894:28201100: pData: 0x8099dfc 0403.158891849:28201100: Hdlr : 0x0 0403.158894084:28201100: Owner: 0x0 0403.158896040:28201100: 0403.158897995:28201100: Command 'actionqueueworkertimeoutthreadshutdown': 0403.158899951:28201100: type : 6 0403.158901907:28201100: pData: 0x8099e00 0403.158905818:28201100: Hdlr : 0x0 0403.158908053:28201100: Owner: 0x0 0403.158910008:28201100: 0403.158911964:28201100: Command 'actionqueueworkerthreadminimummessages': 0403.158913919:28201100: type : 6 0403.158915875:28201100: pData: 0x8099e04 0403.158917831:28201100: Hdlr : 0x0 0403.158920065:28201100: Owner: 0x0 0403.158922021:28201100: 0403.158923977:28201100: Command 'actionqueuemaxfilesize': 0403.158925932:28201100: type : 7 0403.158927888:28201100: pData: 0x8099e08 0403.158932078:28201100: Hdlr : 0x0 0403.158934034:28201100: Owner: 0x0 0403.158935989:28201100: 0403.158937945:28201100: Command 'actionqueuesaveonshutdown': 0403.158939900:28201100: type : 4 0403.158941856:28201100: pData: 0x8099e10 0403.158944091:28201100: Hdlr : 0x0 0403.158946046:28201100: Owner: 0x0 0403.158948002:28201100: 0403.158949957:28201100: Command 'actionqueuedequeueslowdown': 0403.158951913:28201100: type : 6 0403.158953869:28201100: pData: 0x809c6f0 0403.158955824:28201100: Hdlr : 0x0 0403.158960015:28201100: Owner: 0x0 0403.158961970:28201100: 0403.158963926:28201100: Command 'actionqueuedequeuetimebegin': 0403.158965881:28201100: type : 6 0403.158967837:28201100: pData: 0x809c6f4 0403.158970072:28201100: Hdlr : 0x0 0403.158972027:28201100: Owner: 0x0 0403.163339342:28201100: 0403.163341298:28201100: Command 'actionqueuedequeuetimeend': 0403.163343253:28201100: type : 6 0403.163345209:28201100: pData: 0x8099e14 0403.163347164:28201100: Hdlr : 0x0 0403.163349399:28201100: Owner: 0x0 0403.163351355:28201100: 0403.163353310:28201100: 0403.163355266:28201100: Messages with malicious PTR DNS Records are not dropped. 0403.163357221:28201100: Control characters are replaced upon reception. 0403.163359177:28201100: Control character escape sequence prefix is '#'. 0403.163363368:28201100: Main queue size 10000 messages. 0403.163365323:28201100: Main queue worker threads: 1, Perists every 0 updates. 0403.163367279:28201100: Main queue timeouts: shutdown: 0, action completion shutdown: 1000, enq: 2000 0403.163369234:28201100: Main queue watermarks: high: 8000, low: 2000, discard: 9800, discard-severity: 4 0403.163371190:28201100: Main queue save on shutdown 1, max disk space allowed 0 0403.163373425:28201100: Work Directory: '(null)'. 0403.165427876:28201100: logmsg: flags 5, from 'qa-freebsd', msg [origin software="rsyslogd" swVersion="3.17.4" x-pid="9569" x-info="http://www.rsyslog.com"] restart 0403.165806415:28201100: Message has legacy syslog format. 0403.165947774:28201100: Signal 11 (SIGSEGV) occured, execution must be terminated. 0403.166388054:28201100: 0403.166390009:28201100: Recorded Call Order for Thread '28201100': 0403.166391965:28201100: 0: debug.c:1257:dbgClassInit: 0403.166393920:28201100: maximum number of nested calls for this thread: 1. 0403.166396155:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 0403.166398111:28201100: 0403.166400066:28201100: Recorded Call Order for Thread '28201100': 0403.166402022:28201100: 0: syslogd.c:3048:realMain: 0403.166403977:28201100: 1: syslogd.c:2781:mainThread: 0403.166408168:28201100: 2: syslogd.c:2201:init: 0403.166410123:28201100: 3: syslogd.c:894:logmsgInternal: 0403.166412079:28201100: 4: syslogd.c:1598:logmsg: 0403.168448371:28201100: maximum number of nested calls for this thread: 14. 0403.168450327:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 0403.168452282:28201100: 0403.168454238:28201100: Recorded Call Order for Thread 'action 1 queue:Reg/w0 (28201200)': 0403.168456473:28201100: 0: wtp.c:415:wtpWorker: 0403.168458428:28201100: 1: wti.c:403:wtiWorker: 0403.168460384:28201100: maximum number of nested calls for this thread: 4. 0403.168462340:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 0403.168464295:28201100: 0403.168468206:28201100: Recorded Call Order for Thread 'main queue:Reg/w0 (28201300)': 0403.168470441:28201100: 0: wtp.c:415:wtpWorker: 0403.168472397:28201100: 1: wti.c:403:wtiWorker: 0403.168474352:28201100: maximum number of nested calls for this thread: 4. 0403.168476308:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 0403.168478263:28201100: Mutex log for all known mutex operations: 0403.168480219:28201100: mutex 0x28215140 is being waited on by code at wti.c:347, thread action 1 queue:Reg/w0 (28201200) 0403.168482454:28201100: mutex 0x28215130 is being waited on by code at wti.c:347, thread main queue:Reg/w0 (28201300) 0403.168484409:28201100: If the call trace is empty, you may want to ./configure --enable-rtinst 0403.168486365:28201100: To submit bug reports, visit http://www.rsyslog.com/bugs 0403.169922022:28201100: To submit bug reports, visit http://www.rsyslog.com/bugs Abort (core dumped)

-HKS

by **rgerhards** » Fri Jul 11, 2008 10:56 am

which version of freebsd you are using it on? I am asking because I do not see this issue after setting up freebsd. I had hoped it would be a platform-issue, but now it does not look like one...

by **hkspvt** » Fri Jul 11, 2008 3:27 pm

I'm running FreeBSD 7.0. I have replicated this on multiple installs, including a fresh test install with the default options.

I agree that it seems like a platform issue, as I've not been able to replicate it on my (phasing-out) Ubuntu boxes.
-HKS

by **rgerhards** » Fri Jul 11, 2008 3:35 pm

Mmmhhh... It's the same version I am running on. I am not very BSD-literate. Can you tell me how I can update the system to all of the latest packages. Maybe it is a recent package that breaks things... I would really like to see this bug in my lab, as it seems to be hard to do via debug files.

by **rgerhards** » Fri Jul 11, 2008 4:00 pm

Anyhow, I try to get closer to the culprit without a lab repro. Could you download the recent development version (3.19.9) or, even better, be able to pull the current state from git? I am re-evaluating the information you gave and I see where it aborts, but I have no clue yet why. So the appropriate course of action probably is to create a specially instrumented version. I'd like to do that based on the most current build.

by **hkspvt** » Fri Jul 11, 2008 4:01 pm

Upgrading the OS is as simple as:
$ freebsd-upgrade fetch install

There's a lot of different ways to upgrade all your installed packages. Here's my preferred way:

$ pkg_add -r portsnap
$ pkg_add -r portupgrade
$ portsnap fetch extract
$ portupgrade -a

Portsnap updates your ports tree, and portupgrade does exactly what it says. In the future, you can just run "portsnap fetch update" (ie, extract is only necessary the first time it's run).

FWIW, the only package I have installed in my test environment is sudo. Is it possible you have a fix-it package installed that I don't? Perhaps a particular kernel option set?

Also, you're running the FreeBSD 7.0-RELEASE branch, right?

Can you give me the URL to pass to git?

-HKS

by **rgerhards** » Fri Jul 11, 2008 4:07 pm

Thanks for the good information. I'll give it a try. I am running the RELEASE version. You can find rsyslog's git archive at

git://git.adiscon.com/git/rsyslog.git

It would be really cool if you could pull from there. It doesn't include a configure script, so you need to run autoreconf -vfi to generate it.

by **hkspvt** » Fri Jul 11, 2008 5:06 pm

Installed from your git repository with --enable-debug and --enable-rtinst. Here's the debug output. Let me know if you need anything else.

Code: Select all: 1819.947506559:28201100: Writing pidfile /var/run/rsyslogd.pid. 1819.950042635:28201100: rsyslog 3.19.10 - called init() 1819.950044591:28201100: Unloading non-static modules. 1819.950046547:28201100: module lmnet NOT unloaded because it still has a refcount of 3 1819.950048502:28201100: Clearing templates. 1819.950586559:28201100: cfline: '$ActionQueueType LinkedList' 1819.950632096:28201100: action queue type set to LINKEDLIST 1819.950634052:28201100: cfline: '*.* /var/log/debug.log' 1819.951045277:28201100: - traditional PRI filter 1819.951047233:28201100: symbolic name: * ==> 255 1819.951509023:28201100: tried selector action for builtin-file: 0 1819.951510979:28201100: Module builtin-file processed this config line. 1819.951513214:28201100: template: 'RSYSLOG_FileFormat' assigned 1819.951639207:28201100: action 1 queue: save on shutdown 1, max disk space allowed 0 1819.952005734:28201100: action 1 queue: is NOT disk-assisted 1819.952007969:28201100: action 1 queue: type 1, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 1819.952084515:28201100: action 1 queue:Reg: finalizing construction of worker thread pool 1819.952118039:28201100: action 1 queue:Reg/w0: finalizing construction of worker instance data 1819.952119994:28201100: action 1 queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 1819.952532617:28201100: action 1 queue:Reg: high activity - starting 1 additional worker thread(s). 1819.952570052:28201100: action 1 queue:Reg/w0: receiving command 2 1819.953101125:28201100: action 1 queue:Reg: started with state 0, num workers now 1 1819.955039919:28201100: Action 0x28224830: queue 0x2822a260 created 1819.955120376:28201100: selector line successfully processed 1819.955528808:28201100: main queue: is NOT disk-assisted 1819.955531043:28201100: main queue: type 0, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting 1819.955594738:28201100: main queue:Reg: finalizing construction of worker thread pool 1819.955596693:28201100: main queue:Reg/w0: finalizing construction of worker instance data 1819.955598928:28201100: main queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!) 1819.955600884:28201100: main queue:Reg: high activity - starting 1 additional worker thread(s). 1819.955602839:28201100: main queue:Reg/w0: receiving command 2 1819.956145925:28201100: main queue:Reg: started with state 0, num workers now 1 1819.956149836:28201200: action 1 queue:Reg/w0: receiving command 4 1819.957599183:28201100: Main processing queue is initialized and running 1819.957634383:28201100: Active selectors: 1819.958005659:28201100: Selector 1: 1819.958007894:28201100: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Actions: 1819.959088199:28201100: builtin-file: /var/log/debug.log Instance data: 0x2822a140 1819.959145468:28201100: RepeatedMsgReduction: 0 1819.959149379:28201100: Resume Interval: 30 1819.959151335:28201100: Suspended: 0 1819.959574294:28201100: Disabled: 0 1819.959576249:28201100: Exec only when previous is suspended: 0 1819.959578205:28201100: 1819.959580161:28201100: 1819.959582395:28201100: 1819.959584351:28201100: 1819.959586307:28201100: Template: Name='RSYSLOG_DebugFormat' 1819.960107043:28201100: Entry(28221060): type 1, (CONSTANT), value: 'Debug line with all properties: FROMHOST: '' 1819.960504021:28201100: Entry(282210b0): type 2, (FIELD), value: 'fromhost' 1819.960574980:28201100: Entry(28221100): type 1, (CONSTANT), value: '', fromhost-ip: '' 1819.960629456:28201100: Entry(28221150): type 2, (FIELD), value: 'fromhost-ip' 1819.961017773:28201100: Entry(282211a0): type 1, (CONSTANT), value: '', HOSTNAME: '' 1819.961072250:28201100: Entry(282211f0): type 2, (FIELD), value: 'hostname' 1819.961126167:28201100: Entry(28221240): type 1, (CONSTANT), value: '', PRI: ' 1819.961500796:28201100: Entry(28221290): type 2, (FIELD), value: 'pri' 1819.961570916:28201100: Entry(282212e0): type 1, (CONSTANT), value: ', syslogtag '' 1819.961637126:28201100: Entry(28221330): type 2, (FIELD), value: 'syslogtag' 1819.962031031:28201100: Entry(28221380): type 1, (CONSTANT), value: '', programname: '' 1819.962085786:28201100: Entry(282213d0): type 2, (FIELD), value: 'programname' 1819.962139704:28201100: Entry(28221420): type 1, (CONSTANT), value: '', APP-NAME: '' 1819.962558193:28201100: Entry(28221470): type 2, (FIELD), value: 'app-name' 1819.962613228:28201100: Entry(282214c0): type 1, (CONSTANT), value: '', PROCID: '' 1819.963144301:28201100: Entry(28221510): type 2, (FIELD), value: 'procid' 1819.963148212:28201100: Entry(28221560): type 1, (CONSTANT), value: '', MSGID: '' 1819.963596872:28201100: Entry(282215b0): type 2, (FIELD), value: 'msgid' 1819.963651348:28201100: Entry(28221600): type 1, (CONSTANT), value: '', TIMESTAMP: '' 1819.964046650:28201100: Entry(28221650): type 2, (FIELD), value: 'timestamp' 1819.964104199:28201100: Entry(282216a0): type 1, (CONSTANT), value: '', STRUCTURED-DATA: '' 1819.964158955:28201100: Entry(282216f0): type 2, (FIELD), value: 'structured-data' 1819.964547552:28201100: Entry(28221740): type 1, (CONSTANT), value: '', msg: '' 1819.964613761:28201100: Entry(28221790): type 2, (FIELD), value: 'msg' 1819.965001241:28201100: Entry(282217e0): type 1, (CONSTANT), value: '' escaped msg: '' 1819.965068009:28201100: Entry(28221830): type 2, (FIELD), value: 'msg' [drop control-characters] 1819.965135615:28201100: Entry(28221880): type 1, (CONSTANT), value: '' rawmsg: '' 1819.965512758:28201100: Entry(282218d0): type 2, (FIELD), value: 'rawmsg' 1819.965597406:28201100: Entry(28221920): type 1, (CONSTANT), value: '' ' 1819.966002485:28201100: Template: Name='RSYSLOG_SyslogProtocol23Format' 1819.966043272:28201100: Entry(28221970): type 1, (CONSTANT), value: '<' 1819.966098028:28201100: Entry(282219c0): type 2, (FIELD), value: 'pri' 1819.966161444:28201100: Entry(28221a10): type 1, (CONSTANT), value: '>1 ' 1819.966552555:28201100: Entry(28221a60): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 1819.966621000:28201100: Entry(28221ab0): type 1, (CONSTANT), value: ' ' 1819.967007641:28201100: Entry(28221b00): type 2, (FIELD), value: 'hostname' 1819.967062117:28201100: Entry(28221b50): type 1, (CONSTANT), value: ' ' 1819.967116314:28201100: Entry(28221ba0): type 2, (FIELD), value: 'app-name' 1819.967502397:28201100: Entry(28221bf0): type 1, (CONSTANT), value: ' ' 1819.967558828:28201100: Entry(28221c40): type 2, (FIELD), value: 'procid' 1819.968101635:28201100: Entry(28221c90): type 1, (CONSTANT), value: ' ' 1819.968157787:28201100: Entry(28221ce0): type 2, (FIELD), value: 'msgid' 1819.968513977:28201100: Entry(28221d30): type 1, (CONSTANT), value: ' ' 1819.968584378:28201100: Entry(28221d80): type 2, (FIELD), value: 'structured-data' 1819.968638854:28201100: Entry(28221dd0): type 1, (CONSTANT), value: ' ' 1819.969020746:28201100: Entry(28221e20): type 2, (FIELD), value: 'msg' 1819.969084441:28201100: Entry(28221e70): type 1, (CONSTANT), value: ' ' 1819.969150930:28201100: Template: Name='RSYSLOG_FileFormat' 1819.969154841:28201100: Entry(28221ec0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 1819.969592886:28201100: Entry(28221f10): type 1, (CONSTANT), value: ' ' 1819.969649876:28201100: Entry(28221f60): type 2, (FIELD), value: 'hostname' 1819.970034841:28201100: Entry(28221fb0): type 1, (CONSTANT), value: ' ' 1819.970089038:28201100: Entry(28223060): type 2, (FIELD), value: 'syslogtag' 1819.970143794:28201100: Entry(282230b0): type 2, (FIELD), value: 'msg' [drop last LF in msg] 1819.970545800:28201100: Entry(28223100): type 1, (CONSTANT), value: ' ' 1819.970611451:28201100: Template: Name='RSYSLOG_TraditionalFileFormat' 1819.970651679:28201100: Entry(28223150): type 2, (FIELD), value: 'timestamp' 1819.971040556:28201100: Entry(282231a0): type 1, (CONSTANT), value: ' ' 1819.971094752:28201100: Entry(282231f0): type 2, (FIELD), value: 'hostname' 1819.971151464:28201100: Entry(28223240): type 1, (CONSTANT), value: ' ' 1819.971155654:28201100: Entry(28223290): type 2, (FIELD), value: 'syslogtag' 1819.971590067:28201100: Entry(282232e0): type 2, (FIELD), value: 'msg' [drop last LF in msg] 1819.972002968:28201100: Entry(28223330): type 1, (CONSTANT), value: ' ' 1819.972069457:28201100: Template: Name=' WallFmt' 1819.972109406:28201100: Entry(28223380): type 1, (CONSTANT), value: ' Message from syslogd@' 1819.972511134:28201100: Entry(282233d0): type 2, (FIELD), value: 'hostname' 1819.973580264:28201100: Entry(28223420): type 1, (CONSTANT), value: ' at ' 1819.973637534:28201100: Entry(28223470): type 2, (FIELD), value: 'timegenerated' 1819.974026410:28201100: Entry(282234c0): type 1, (CONSTANT), value: ' ... ' 1819.974093458:28201100: Entry(28223510): type 2, (FIELD), value: 'syslogtag' 1819.974147934:28201100: Entry(28223560): type 2, (FIELD), value: 'msg' 1819.974151845:28201100: Entry(282235b0): type 1, (CONSTANT), value: ' ' 1819.974613356:28201100: Template: Name='RSYSLOG_ForwardFormat' 1819.974654143:28201100: Entry(28223600): type 1, (CONSTANT), value: '<' 1819.975083528:28201100: Entry(28223650): type 2, (FIELD), value: 'pri' 1819.975141636:28201100: Entry(282236a0): type 1, (CONSTANT), value: '>' 1819.975145547:28201100: Entry(282236f0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date] 1819.975604264:28201100: Entry(28223740): type 1, (CONSTANT), value: ' ' 1819.975658740:28201100: Entry(28223790): type 2, (FIELD), value: 'hostname' 1819.976045941:28201100: Entry(282237e0): type 1, (CONSTANT), value: ' ' 1819.976100137:28201100: Entry(28223830): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 1819.976502982:28201100: Entry(28223880): type 2, (FIELD), value: 'msg' 1819.976559693:28201100: Template: Name='RSYSLOG_TraditionalForwardFormat' 1819.976601598:28201100: Entry(282238d0): type 1, (CONSTANT), value: '<' 1819.976657191:28201100: Entry(28223920): type 2, (FIELD), value: 'pri' 1819.977046347:28201100: Entry(28223970): type 1, (CONSTANT), value: '>' 1819.977100823:28201100: Entry(282239c0): type 2, (FIELD), value: 'timestamp' 1819.977154741:28201100: Entry(28223a10): type 1, (CONSTANT), value: ' ' 1819.977553674:28201100: Entry(28223a60): type 2, (FIELD), value: 'hostname' 1819.977608429:28201100: Entry(28223ab0): type 1, (CONSTANT), value: ' ' 1819.977662626:28201100: Entry(28223b00): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32] 1819.978069661:28201100: Entry(28223b50): type 2, (FIELD), value: 'msg' 1819.979127058:28201100: Template: Name=' StdUsrMsgFmt' 1819.979168684:28201100: Entry(28223ba0): type 1, (CONSTANT), value: ' ' 1819.979172874:28201100: Entry(28223bf0): type 2, (FIELD), value: 'syslogtag' 1819.979597509:28201100: Entry(28223c40): type 2, (FIELD), value: 'msg' 1819.979651706:28201100: Entry(28223c90): type 1, (CONSTANT), value: ' ' 1819.980049522:28201100: Template: Name=' StdDBFmt' [SQL-Format (MySQL)] 1819.980103160:28201100: Entry(28223ce0): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 1819.980504328:28201100: Entry(28223d30): type 2, (FIELD), value: 'msg' 1819.980574728:28201100: Entry(28223d80): type 1, (CONSTANT), value: '', ' 1819.980628646:28201100: Entry(28223dd0): type 2, (FIELD), value: 'syslogfacility' 1819.981016963:28201100: Entry(28223e20): type 1, (CONSTANT), value: ', '' 1819.981071160:28201100: Entry(28223e70): type 2, (FIELD), value: 'hostname' 1819.981125357:28201100: Entry(28223ec0): type 1, (CONSTANT), value: '', ' 1819.981502779:28201100: Entry(28223f10): type 2, (FIELD), value: 'syslogpriority' 1819.981574017:28201100: Entry(28223f60): type 1, (CONSTANT), value: ', '' 1819.981628214:28201100: Entry(28223fb0): type 2, (FIELD), value: 'timereported' [Format as MySQL-Date] 1819.982032176:28201100: Entry(28224060): type 1, (CONSTANT), value: '', '' 1819.982086932:28201100: Entry(282240b0): type 2, (FIELD), value: 'timegenerated' [Format as MySQL-Date] 1819.982155655:28201100: Entry(28224100): type 1, (CONSTANT), value: '', ' 1819.982559897:28201100: Entry(28224150): type 2, (FIELD), value: 'iut' 1819.982843173:28201100: Entry(282241a0): type 1, (CONSTANT), value: ', '' 1819.983006881:28201100: Entry(282241f0): type 2, (FIELD), value: 'syslogtag' 1819.983061637:28201100: Entry(28224240): type 1, (CONSTANT), value: '')' 1819.983128405:28201100: Template: Name=' StdPgSQLFmt' [SQL-Format (standard SQL)] 1819.984567414:28201100: Entry(28224290): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('' 1819.984627199:28201100: Entry(282242e0): type 2, (FIELD), value: 'msg' 1819.985016913:28201100: Entry(28224330): type 1, (CONSTANT), value: '', ' 1819.985071948:28201100: Entry(28224380): type 2, (FIELD), value: 'syslogfacility' 1819.985126424:28201100: Entry(282243d0): type 1, (CONSTANT), value: ', '' 1819.985502450:28201100: Entry(28224420): type 2, (FIELD), value: 'hostname' 1819.985572011:28201100: Entry(28224470): type 1, (CONSTANT), value: '', ' 1819.985626208:28201100: Entry(282244c0): type 2, (FIELD), value: 'syslogpriority' 1819.986015643:28201100: Entry(28224510): type 1, (CONSTANT), value: ', '' 1819.986079338:28201100: Entry(28224560): type 2, (FIELD), value: 'timereported' [Format as PgSQL-Date] 1819.986558170:28201100: Entry(282245b0): type 1, (CONSTANT), value: '', '' 1819.986615440:28201100: Entry(28224600): type 2, (FIELD), value: 'timegenerated' [Format as PgSQL-Date] 1819.987014373:28201100: Entry(28224650): type 1, (CONSTANT), value: '', ' 1819.987070526:28201100: Entry(282246a0): type 2, (FIELD), value: 'iut' 1819.987125561:28201100: Entry(282246f0): type 1, (CONSTANT), value: ', '' 1819.987509129:28201100: Entry(28224740): type 2, (FIELD), value: 'syslogtag' 1819.987566678:28201100: Entry(28224790): type 1, (CONSTANT), value: '')' 1819.987622272:28201100: Loaded Module: Name='lmnet', IFVersion=4, type=library module. 1819.988026234:28201100: Entry points: 1819.988028190:28201100: queryEtryPt: 0x281f7660 1819.988030145:28201100: doAction: 0x0 1819.988032380:28201100: parseSelectorAct: 0x0 1819.988034336:28201100: dbgPrintInstInfo: 0x0 1819.988038247:28201100: freeInstance: 0x0 1819.988040202:28201100: 1819.988042158:28201100: Loaded Module: Name='builtin-file', IFVersion=4, type=output module. 1819.990187961:28201100: Entry points: 1819.990189917:28201100: queryEtryPt: 0x80578f0 1819.990194107:28201100: doAction: 0x8058c80 1819.990196063:28201100: parseSelectorAct: 0x80580b0 1819.990198018:28201100: dbgPrintInstInfo: 0x8058a30 1819.990199974:28201100: freeInstance: 0x8057f60 1819.990201930:28201100: 1819.990203885:28201100: Loaded Module: Name='builtin-fwd', IFVersion=4, type=output module. 1819.991128304:28201100: Entry points: 1819.991130260:28201100: queryEtryPt: 0x8055c30 1819.991134171:28201100: doAction: 0x8056f70 1819.991136406:28201100: parseSelectorAct: 0x80562d0 1819.991138361:28201100: dbgPrintInstInfo: 0x8056260 1819.991140317:28201100: freeInstance: 0x8056120 1819.991142272:28201100: 1819.991144228:28201100: Loaded Module: Name='builtin-shell', IFVersion=4, type=output module. 1819.992080939:28201100: Entry points: 1819.992082895:28201100: queryEtryPt: 0x80545c0 1819.992084850:28201100: doAction: 0x8054ab0 1819.993193092:28201100: parseSelectorAct: 0x80547f0 1819.993195047:28201100: dbgPrintInstInfo: 0x8054a40 1819.993197003:28201100: freeInstance: 0x8054780 1819.993198958:28201100: 1819.993201193:28201100: Loaded Module: Name='builtin-discard', IFVersion=4, type=output module. 1819.993738413:28201100: Entry points: 1819.993740368:28201100: queryEtryPt: 0x80595d0 1819.993742324:28201100: doAction: 0x8059790 1819.993744279:28201100: parseSelectorAct: 0x8059860 1819.993746235:28201100: dbgPrintInstInfo: 0x8059580 1819.993748470:28201100: freeInstance: 0x80597f0 1819.993750425:28201100: 1819.993752381:28201100: Loaded Module: Name='builtin-usrmsg', IFVersion=4, type=output module. 1819.994242946:28201100: Entry points: 1819.994244901:28201100: queryEtryPt: 0x8054e00 1819.994246857:28201100: doAction: 0x8055500 1819.994248813:28201100: parseSelectorAct: 0x80550c0 1819.994250768:28201100: dbgPrintInstInfo: 0x8054fc0 1819.963011323:action 1 queue:Reg/w0: action 1 queue: entering rate limiter 1819.998571150:28201300: main queue:Reg/w0: receiving command 4 1819.998641550:28201100: freeInstance: 0x8055050 1819.998643505:28201100: 1819.998645461:28201100: Sytem Line Configuration Commands: 1819.998647416:28201100: Command 'workdirectory': 1819.998649651:28201100: type : 11 1819.998651607:28201100: pData: 0x80a44f0 1819.998653562:28201100: Hdlr : 0x0 1819.998657473:28201100: Owner: 0x0 1819.998659429:28201100: 1819.999607594:28201100: Command 'dropmsgswithmaliciousdnsptrrecords': 1819.999609829:28201100: type : 4 1819.999611785:28201100: pData: 0x80a4500 1819.999613740:28201100: Hdlr : 0x0 1819.999615696:28201100: Owner: 0x0 1819.999617651:28201100: 1819.999619607:28201100: Command 'defaultnetstreamdriver': 1819.999621562:28201100: type : 11 1819.999623797:28201100: pData: 0x80a44e0 1819.999627708:28201100: Hdlr : 0x0 1819.999629664:28201100: Owner: 0x0 1819.999631620:28201100: 1819.999633575:28201100: Command 'defaultnetstreamdrivercafile': 1819.999635810:28201100: type : 11 1819.999637766:28201100: pData: 0x80a44e4 1819.999639721:28201100: Hdlr : 0x0 1819.999641677:28201100: Owner: 0x0 1819.999643632:28201100: 1819.999645588:28201100: Command 'defaultnetstreamdriverkeyfile': 1819.999647823:28201100: type : 11 1819.999649778:28201100: pData: 0x80a44e8 1819.999653690:28201100: Hdlr : 0x0 1819.999655645:28201100: Owner: 0x0 1819.999657601:28201100: 1819.999659556:28201100: Command 'defaultnetstreamdrivercertfile': 1819.999661791:28201100: type : 11 1819.999663747:28201100: pData: 0x80a44ec 1819.999665702:28201100: Hdlr : 0x0 1819.999667658:28201100: Owner: 0x0 1819.999669613:28201100: 1819.999671569:28201100: Command 'resetconfigvariables': 1819.999673804:28201100: type : 1 1819.999675759:28201100: pData: 0x0 1819.999679670:28201100: Hdlr : 0x805a940 1819.999681626:28201100: Owner: 0x0 1819.999683581:28201100: 1819.999685816:28201100: type : 1 1819.999687772:28201100: pData: 0x0 1819.999689728:28201100: Hdlr : 0x8057db0 1820.003080382:28201100: Owner: 0x8057d50 1820.003082617:28201100: 1820.003084572:28201100: type : 1 1820.003086528:28201100: pData: 0x0 1820.003090439:28201100: Hdlr : 0x8055f30 1820.003092395:28201100: Owner: 0x8055f60 1820.003094630:28201100: 1820.003096585:28201100: type : 1 1820.003098541:28201100: pData: 0x0 1820.003100496:28201100: Hdlr : 0x804ee10 1820.003102452:28201100: Owner: 0x0 1820.003104407:28201100: 1820.004415747:28201100: Command 'dynafilecachesize': 1820.004417982:28201100: type : 6 1820.004419938:28201100: pData: 0x0 1820.004421893:28201100: Hdlr : 0x8057ab0 1820.004423849:28201100: Owner: 0x8057d50 1820.004425804:28201100: 1820.004427760:28201100: Command 'dirowner': 1820.004429995:28201100: type : 2 1820.004431950:28201100: pData: 0x80a4464 1820.004433906:28201100: Hdlr : 0x0 1820.004435861:28201100: Owner: 0x8057d50 1820.004439773:28201100: 1820.004442007:28201100: Command 'dirgroup': 1820.004443963:28201100: type : 3 1820.004445919:28201100: pData: 0x80a4468 1820.004447874:28201100: Hdlr : 0x0 1820.004449830:28201100: Owner: 0x8057d50 1820.004451785:28201100: 1820.004453741:28201100: Command 'fileowner': 1820.004455976:28201100: type : 2 1820.004457931:28201100: pData: 0x80a445c 1820.004459887:28201100: Hdlr : 0x0 1820.004461842:28201100: Owner: 0x8057d50 1820.004465754:28201100: 1820.004467988:28201100: Command 'filegroup': 1820.004469944:28201100: type : 3 1820.004471900:28201100: pData: 0x80a4460 1820.004473855:28201100: Hdlr : 0x0 1820.004475811:28201100: Owner: 0x8057d50 1820.004477766:28201100: 1820.004480001:28201100: Command 'dircreatemode': 1820.004481957:28201100: type : 5 1820.004483912:28201100: pData: 0x80a1c70 1820.004485868:28201100: Hdlr : 0x0 1820.004487823:28201100: Owner: 0x8057d50 1820.004492014:28201100: 1820.004493969:28201100: Command 'filecreatemode': 1820.004495925:28201100: type : 5 1820.004497881:28201100: pData: 0x80a1c74 1820.006628319:main queue:Reg/w0: main queue: entering rate limiter 1820.010088256:action 1 queue:Reg/w0: action 1 queue:Reg/w0: worker IDLE, waiting for work. 1820.010139659:28201100: Hdlr : 0x0 1820.010141615:28201100: Owner: 0x8057d50 1820.010143849:28201100: 1820.010145805:28201100: Command 'createdirs': 1820.010147761:28201100: type : 4 1820.010149716:28201100: pData: 0x80a446c 1820.010151672:28201100: Hdlr : 0x0 1820.010153627:28201100: Owner: 0x8057d50 1820.010155583:28201100: 1820.010157818:28201100: Command 'failonchownfailure': 1820.010159773:28201100: type : 4 1820.010161729:28201100: pData: 0x80a4458 1820.010163684:28201100: Hdlr : 0x0 1820.010165640:28201100: Owner: 0x8057d50 1820.010169830:28201100: 1820.010171786:28201100: Command 'actionfileenablesync': 1820.010173741:28201100: type : 4 1820.010175697:28201100: pData: 0x80a4410 1820.010177653:28201100: Hdlr : 0x0 1820.010179608:28201100: Owner: 0x8057d50 1820.010181843:28201100: 1820.010183799:28201100: Command 'actionfiledefaulttemplate': 1820.010185754:28201100: type : 11 1820.010187710:28201100: pData: 0x80a4414 1820.010189665:28201100: Hdlr : 0x0 1820.012542758:28201100: Owner: 0x0 1820.012544713:28201100: 1820.012546669:28201100: Command 'actionforwarddefaulttemplate': 1820.012548625:28201100: type : 11 1820.012550859:28201100: pData: 0x80a43b0 1820.012552815:28201100: Hdlr : 0x0 1820.012554771:28201100: Owner: 0x0 1820.012556726:28201100: 1820.012558682:28201100: Command 'actionsendstreamdriver': 1820.012560637:28201100: type : 11 1820.012564828:28201100: pData: 0x80a43b4 1820.012566783:28201100: Hdlr : 0x0 1820.012568739:28201100: Owner: 0x0 1820.012570694:28201100: 1820.012572650:28201100: Command 'actionsendstreamdrivermode': 1820.012574885:28201100: type : 6 1820.012576840:28201100: pData: 0x80a43b8 1820.012578796:28201100: Hdlr : 0x0 1820.012580751:28201100: Owner: 0x0 1820.012582707:28201100: 1820.012584662:28201100: Command 'actionsendstreamdriverauthmode': 1820.012586897:28201100: type : 11 1820.014582682:28201100: pData: 0x80a43bc 1820.014584637:28201100: Hdlr : 0x0 1820.014586593:28201100: Owner: 0x0 1820.014588549:28201100: 1820.014590504:28201100: Command 'actionsendstreamdriverpermittedpeer': 1820.014592460:28201100: type : 11 1820.014594415:28201100: pData: 0x0 1820.014596650:28201100: Hdlr : 0x80560a0 1820.014598606:28201100: Owner: 0x0 1820.014600561:28201100: 1820.014604472:28201100: Command 'actionsendresendlastmsgonreconnect': 1820.014606428:28201100: type : 4 1820.014608663:28201100: pData: 0x80a43c0 1820.014610618:28201100: Hdlr : 0x0 1820.014612574:28201100: Owner: 0x0 1820.014614529:28201100: 1820.014616485:28201100: Command 'actionresumeretrycount': 1820.014618440:28201100: type : 6 1820.014620675:28201100: pData: 0x80a5ab0 1820.014622631:28201100: Hdlr : 0x0 1820.014624587:28201100: Owner: 0x0 1820.014626542:28201100: 1820.014630453:28201100: Command 'mainmsgqueuefilename': 1820.014632688:28201100: type : 11 1820.014634644:28201100: pData: 0x80a4260 1820.014636599:28201100: Hdlr : 0x0 1820.014638555:28201100: Owner: 0x0 1820.017020422:28201100: 1820.017022377:28201100: Command 'mainmsgqueuesize': 1820.017024612:28201100: type : 6 1820.017026568:28201100: pData: 0x80a12e0 1820.017028523:28201100: Hdlr : 0x0 1820.017030479:28201100: Owner: 0x0 1820.017034390:28201100: 1820.017036625:28201100: Command 'mainmsgqueuehighwatermark': 1820.017038580:28201100: type : 6 1820.017040536:28201100: pData: 0x80a12e4 1820.017042492:28201100: Hdlr : 0x0 1820.017044447:28201100: Owner: 0x0 1820.017046403:28201100: 1820.017048358:28201100: Command 'mainmsgqueuelowwatermark': 1820.017050593:28201100: type : 6 1820.017052549:28201100: pData: 0x80a12e8 1820.017054504:28201100: Hdlr : 0x0 1820.017056460:28201100: Owner: 0x0 1820.017060371:28201100: 1820.017062606:28201100: Command 'mainmsgqueuediscardmark': 1820.017064562:28201100: type : 6 1820.017066517:28201100: pData: 0x80a12ec 1820.021095521:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1820.021103622:28201100: Hdlr : 0x0 1820.021105578:28201100: Owner: 0x0 1820.021107533:28201100: 1820.021109489:28201100: Command 'mainmsgqueuediscardseverity': 1820.021111445:28201100: type : 10 1820.021113400:28201100: pData: 0x80a12f0 1820.021115356:28201100: Hdlr : 0x0 1820.021117591:28201100: Owner: 0x0 1820.021119546:28201100: 1820.021121502:28201100: Command 'mainmsgqueuecheckpointinterval': 1820.021123457:28201100: type : 6 1820.021125413:28201100: pData: 0x80a4264 1820.021129603:28201100: Hdlr : 0x0 1820.021131559:28201100: Owner: 0x0 1820.021133514:28201100: 1820.021135470:28201100: Command 'mainmsgqueuetype': 1820.021137425:28201100: type : 11 1820.021139381:28201100: pData: 0x0 1820.021141616:28201100: Hdlr : 0x804f4c0 1820.021143571:28201100: Owner: 0x0 1820.021145527:28201100: 1820.021147483:28201100: Command 'mainmsgqueueworkerthreads': 1820.021149438:28201100: type : 6 1820.021151394:28201100: pData: 0x80a12f4 1820.021155584:28201100: Hdlr : 0x0 1820.021157540:28201100: Owner: 0x0 1820.021159495:28201100: 1820.021161451:28201100: Command 'mainmsgqueuetimeoutshutdown': 1820.021163407:28201100: type : 6 1820.021165362:28201100: pData: 0x80a4268 1820.021167597:28201100: Hdlr : 0x0 1820.021169553:28201100: Owner: 0x0 1820.021171508:28201100: 1820.021173464:28201100: Command 'mainmsgqueuetimeoutactioncompletion': 1820.021175419:28201100: type : 6 1820.021177375:28201100: pData: 0x80a12f8 1820.021181565:28201100: Hdlr : 0x0 1820.021183521:28201100: Owner: 0x0 1820.021185476:28201100: 1820.021187432:28201100: Command 'mainmsgqueuetimeoutenqueue': 1820.021189387:28201100: type : 6 1820.021191622:28201100: pData: 0x80a12fc 1820.021193578:28201100: Hdlr : 0x0 1820.021195533:28201100: Owner: 0x0 1820.021199445:28201100: 1820.021201400:28201100: Command 'mainmsgqueueworketimeoutrthreadshutdown': 1820.021203356:28201100: type : 6 1820.025656436:28201100: pData: 0x80a1300 1820.025658391:28201100: Hdlr : 0x0 1820.025660347:28201100: Owner: 0x0 1820.025662302:28201100: 1820.025664258:28201100: Command 'mainmsgqueuedequeueslowdown': 1820.025666213:28201100: type : 6 1820.025668448:28201100: pData: 0x80a426c 1820.025672359:28201100: Hdlr : 0x0 1820.025674315:28201100: Owner: 0x0 1820.025676270:28201100: 1820.025678226:28201100: Command 'mainmsgqueueworkerthreadminimummessages': 1820.025680461:28201100: type : 6 1820.025682416:28201100: pData: 0x80a1304 1820.025684372:28201100: Hdlr : 0x0 1820.025686328:28201100: Owner: 0x0 1820.025688283:28201100: 1820.025690239:28201100: Command 'mainmsgqueuemaxfilesize': 1820.025692194:28201100: type : 7 1820.025694429:28201100: pData: 0x80a1308 1820.025698340:28201100: Hdlr : 0x0 1820.025700296:28201100: Owner: 0x0 1820.025702252:28201100: 1820.025704207:28201100: Command 'mainmsgqueuemaxdiskspace': 1820.025706442:28201100: type : 7 1820.025708398:28201100: pData: 0x80a4270 1820.025710353:28201100: Hdlr : 0x0 1820.025712309:28201100: Owner: 0x0 1820.025714264:28201100: 1820.025716220:28201100: Command 'mainmsgqueuesaveonshutdown': 1820.025718455:28201100: type : 4 1820.025720410:28201100: pData: 0x80a1310 1820.025724321:28201100: Hdlr : 0x0 1820.025726277:28201100: Owner: 0x0 1820.025728232:28201100: 1820.025730467:28201100: Command 'mainmsgqueuedequeuetimebegin': 1820.025732423:28201100: type : 6 1820.025734378:28201100: pData: 0x80a4278 1820.025736334:28201100: Hdlr : 0x0 1820.025738290:28201100: Owner: 0x0 1820.025740245:28201100: 1820.025742201:28201100: Command 'mainmsgqueuedequeuetimeend': 1820.025744436:28201100: type : 6 1820.025746391:28201100: pData: 0x80a1314 1820.025748347:28201100: Hdlr : 0x0 1820.025752258:28201100: Owner: 0x0 1820.025754213:28201100: 1820.025756448:28201100: Command 'repeatedmsgreduction': 1820.025758404:28201100: type : 4 1820.025760359:28201100: pData: 0x80a5b9c 1820.030503141:28201100: Hdlr : 0x0 1820.030505097:28201100: Owner: 0x0 1820.030507052:28201100: 1820.030509008:28201100: Command 'actionexeconlywhenpreviousissuspended': 1820.030510963:28201100: type : 4 1820.030513198:28201100: pData: 0x80a5ba0 1820.030515154:28201100: Hdlr : 0x0 1820.030517109:28201100: Owner: 0x0 1820.030519065:28201100: 1820.030521020:28201100: Command 'actionexeconlyonceeveryinterval': 1820.030522976:28201100: type : 6 1820.030525211:28201100: pData: 0x80a423c 1820.030529122:28201100: Hdlr : 0x0 1820.030531077:28201100: Owner: 0x0 1820.030533033:28201100: 1820.030534989:28201100: Command 'actionresumeinterval': 1820.030537223:28201100: type : 6 1820.030539179:28201100: pData: 0x0 1820.030541135:28201100: Hdlr : 0x804f4b0 1820.030543090:28201100: Owner: 0x0 1820.030545046:28201100: 1820.030547001:28201100: Command 'controlcharacterescapeprefix': 1820.030548957:28201100: type : 8 1820.030551192:28201100: pData: 0x80a1318 1820.030553147:28201100: Hdlr : 0x0 1820.032545300:28201100: Owner: 0x0 1820.032547255:28201100: 1820.032549211:28201100: Command 'escapecontrolcharactersonreceive': 1820.032551446:28201100: type : 4 1820.032553402:28201100: pData: 0x80a131c 1820.032555357:28201100: Hdlr : 0x0 1820.032557313:28201100: Owner: 0x0 1820.032559268:28201100: 1820.032561224:28201100: Command 'droptrailinglfonreception': 1820.032563179:28201100: type : 4 1820.032565414:28201100: pData: 0x80a1320 1820.032569325:28201100: Hdlr : 0x0 1820.032571281:28201100: Owner: 0x0 1820.032573237:28201100: 1820.032575192:28201100: Command 'template': 1820.032577427:28201100: type : 1 1820.032579383:28201100: pData: 0x0 1820.032581338:28201100: Hdlr : 0x805d060 1820.032583294:28201100: Owner: 0x0 1820.032585249:28201100: 1820.032587205:28201100: Command 'outchannel': 1820.032589440:28201100: type : 1 1820.032591395:28201100: pData: 0x1 1820.032595306:28201100: Hdlr : 0x805d060 1820.035503776:28201100: Owner: 0x0 1820.035505732:28201100: 1820.035507688:28201100: Command 'allowedsender': 1820.035509643:28201100: type : 1 1820.035511599:28201100: pData: 0x2 1820.035513554:28201100: Hdlr : 0x805d060 1820.035515789:28201100: Owner: 0x0 1820.035517745:28201100: 1820.035519700:28201100: Command 'modload': 1820.035521656:28201100: type : 1 1820.035525567:28201100: pData: 0x0 1820.035527802:28201100: Hdlr : 0x805d210 1820.035529758:28201100: Owner: 0x0 1820.035531713:28201100: 1820.035533669:28201100: Command 'includeconfig': 1820.035535624:28201100: type : 1 1820.035537580:28201100: pData: 0x0 1820.035539535:28201100: Hdlr : 0x805d3a0 1820.035541770:28201100: Owner: 0x0 1820.035543726:28201100: 1820.035545681:28201100: Command 'umask': 1820.035547637:28201100: type : 5 1820.035549592:28201100: pData: 0x0 1820.035553783:28201100: Hdlr : 0x804f480 1820.035555738:28201100: Owner: 0x0 1820.035557694:28201100: 1820.035559650:28201100: Command 'debugprinttemplatelist': 1820.035561605:28201100: type : 4 1820.035563561:28201100: pData: 0x80a1324 1820.035565796:28201100: Hdlr : 0x0 1820.035567751:28201100: Owner: 0x0 1820.035569707:28201100: 1820.035571662:28201100: Command 'debugprintmodulelist': 1820.035573618:28201100: type : 4 1820.035575573:28201100: pData: 0x80a1328 1820.035579764:28201100: Hdlr : 0x0 1820.035581719:28201100: Owner: 0x0 1820.035583675:28201100: 1820.035585630:28201100: Command 'debugprintcfsyslinehandlerlist': 1820.035587586:28201100: type : 4 1820.035589542:28201100: pData: 0x80a132c 1820.035591776:28201100: Hdlr : 0x0 1820.035593732:28201100: Owner: 0x0 1820.035595688:28201100: 1820.035597643:28201100: Command 'moddir': 1820.035599599:28201100: type : 11 1820.035601554:28201100: pData: 0x80a5610 1820.035605745:28201100: Hdlr : 0x0 1820.035607700:28201100: Owner: 0x0 1820.035609656:28201100: 1820.035611612:28201100: Command 'actionqueuefilename': 1820.035613567:28201100: type : 11 1820.040628171:28201100: pData: 0x80a5abc 1820.040630126:28201100: Hdlr : 0x0 1820.040632082:28201100: Owner: 0x0 1820.040634317:28201100: 1820.040636272:28201100: Command 'actionqueuesize': 1820.040638228:28201100: type : 6 1820.040640183:28201100: pData: 0x80a2fe0 1820.040642139:28201100: Hdlr : 0x0 1820.040644095:28201100: Owner: 0x0 1820.040648285:28201100: 1820.040650241:28201100: Command 'actionqueuemaxdiskspace': 1820.040652196:28201100: type : 7 1820.040654152:28201100: pData: 0x80a5ac0 1820.040656107:28201100: Hdlr : 0x0 1820.040658342:28201100: Owner: 0x0 1820.040660298:28201100: 1820.040662253:28201100: Command 'actionqueuehighwatermark': 1820.040664209:28201100: type : 6 1820.040666165:28201100: pData: 0x80a2fe4 1820.040668120:28201100: Hdlr : 0x0 1820.040670076:28201100: Owner: 0x0 1820.040672311:28201100: 1820.042541822:28201100: Command 'actionqueuelowwatermark': 1820.042544057:28201100: type : 6 1820.042546012:28201100: pData: 0x80a2fe8 1820.042547968:28201100: Hdlr : 0x0 1820.042549923:28201100: Owner: 0x0 1820.042551879:28201100: 1820.042553835:28201100: Command 'actionqueuediscardmark': 1820.042556070:28201100: type : 6 1820.042558025:28201100: pData: 0x80a2fec 1820.043502558:28201100: Hdlr : 0x0 1820.043504514:28201100: Owner: 0x0 1820.043506749:28201100: 1820.043508704:28201100: Command 'actionqueuediscardseverity': 1820.043510660:28201100: type : 6 1820.043512616:28201100: pData: 0x80a2ff0 1820.043514571:28201100: Hdlr : 0x0 1820.043516527:28201100: Owner: 0x0 1820.043518762:28201100: 1820.043520717:28201100: Command 'actionqueuecheckpointinterval': 1820.043522673:28201100: type : 6 1820.043524628:28201100: pData: 0x80a5ac8 1820.043528540:28201100: Hdlr : 0x0 1820.043530495:28201100: Owner: 0x0 1820.043532730:28201100: 1820.043534686:28201100: Command 'actionqueuetype': 1820.043536641:28201100: type : 11 1820.043538597:28201100: pData: 0x0 1820.043540552:28201100: Hdlr : 0x808e1b0 1820.046022711:28201100: Owner: 0x0 1820.046024667:28201100: 1820.046026622:28201100: Command 'actionqueueworkerthreads': 1820.046028578:28201100: type : 6 1820.046030533:28201100: pData: 0x80a2ff4 1820.046032489:28201100: Hdlr : 0x0 1820.046036679:28201100: Owner: 0x0 1820.046038635:28201100: 1820.046040591:28201100: Command 'actionqueuetimeoutshutdown': 1820.046042546:28201100: type : 6 1820.046044502:28201100: pData: 0x80a5acc 1820.046046737:28201100: Hdlr : 0x0 1820.046048692:28201100: Owner: 0x0 1820.046050648:28201100: 1820.046052603:28201100: Command 'actionqueuetimeoutactioncompletion': 1820.046054559:28201100: type : 6 1820.046056514:28201100: pData: 0x80a2ff8 1820.046058749:28201100: Hdlr : 0x0 1820.046060705:28201100: Owner: 0x0 1820.046062660:28201100: 1820.046064616:28201100: Command 'actionqueuetimeoutenqueue': 1820.046066571:28201100: type : 6 1820.046068527:28201100: pData: 0x80a2ffc 1820.046070483:28201100: Hdlr : 0x0 1820.046072717:28201100: Owner: 0x0 1820.046074673:28201100: 1820.046076629:28201100: Command 'actionqueueworkertimeoutthreadshutdown': 1820.046078584:28201100: type : 6 1820.046080540:28201100: pData: 0x80a3000 1820.046082495:28201100: Hdlr : 0x0 1820.046084730:28201100: Owner: 0x0 1820.046086686:28201100: 1820.046088641:28201100: Command 'actionqueueworkerthreadminimummessages': 1820.046090597:28201100: type : 6 1820.046092553:28201100: pData: 0x80a3004 1820.046094508:28201100: Hdlr : 0x0 1820.046096743:28201100: Owner: 0x0 1820.046100654:28201100: 1820.046102610:28201100: Command 'actionqueuemaxfilesize': 1820.046104565:28201100: type : 7 1820.046106521:28201100: pData: 0x80a3008 1820.046108476:28201100: Hdlr : 0x0 1820.046110711:28201100: Owner: 0x0 1820.046112667:28201100: 1820.046114622:28201100: Command 'actionqueuesaveonshutdown': 1820.046116578:28201100: type : 4 1820.046118533:28201100: pData: 0x80a3010 1820.046120489:28201100: Hdlr : 0x0 1820.049128413:28201100: Owner: 0x0 1820.049130369:28201100: 1820.049132604:28201100: Command 'actionqueuedequeueslowdown': 1820.049136515:28201100: type : 6 1820.049138470:28201100: pData: 0x80a5ad0 1820.049140426:28201100: Hdlr : 0x0 1820.049142381:28201100: Owner: 0x0 1820.049144616:28201100: 1820.049146572:28201100: Command 'actionqueuedequeuetimebegin': 1820.049148527:28201100: type : 6 1820.049150483:28201100: pData: 0x80a5ad4 1820.049152438:28201100: Hdlr : 0x0 1820.049154394:28201100: Owner: 0x0 1820.049156629:28201100: 1820.049158585:28201100: Command 'actionqueuedequeuetimeend': 1820.049162496:28201100: type : 6 1820.049164451:28201100: pData: 0x80a3014 1820.049166407:28201100: Hdlr : 0x0 1820.049168362:28201100: Owner: 0x0 1820.049170597:28201100: 1820.049172553:28201100: 1820.049174509:28201100: Messages with malicious PTR DNS Records are not dropped. 1820.049176464:28201100: Control characters are replaced upon reception. 1820.049178420:28201100: Control character escape sequence prefix is '#'. 1820.049180375:28201100: Main queue size 10000 messages. 1820.049182610:28201100: Main queue worker threads: 1, Perists every 0 updates. 1820.049184566:28201100: Main queue timeouts: shutdown: 0, action completion shutdown: 1000, enq: 2000 1820.049188477:28201100: Main queue watermarks: high: 8000, low: 2000, discard: 9800, discard-severity: 4 1820.049190432:28201100: Main queue save on shutdown 1, max disk space allowed 0 1820.049192388:28201100: Work Directory: ''. 1820.052533036:28201100: logmsg: flags 5, pri 7, from 'qa-freebsd', msg [origin software="rsyslogd" swVersion="3.19.10" x-pid="30999" x-info="http://www.rsyslog.com"] restart 1820.052534991:28201100: Message has legacy syslog format. 1820.052662382:28201100: Signal 11 (SIGSEGV) occured, execution must be terminated. 1820.053133392:28201100: 1820.053135347:28201100: Recorded Call Order for Thread '28201100': 1820.053139258:28201100: 0: debug.c:1260:dbgClassInit: 1820.053141214:28201100: maximum number of nested calls for this thread: 1. 1820.054558433:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 1820.054560668:28201100: 1820.054562623:28201100: Recorded Call Order for Thread '28201100': 1820.054564579:28201100: 0: syslogd.c:2975:realMain: 1820.054566534:28201100: 1: syslogd.c:2739:mainThread: 1820.054568490:28201100: 2: syslogd.c:2197:init: 1820.054570446:28201100: 3: syslogd.c:887:logmsgInternal: 1820.054572401:28201100: 4: syslogd.c:1600:logmsg: 1820.054574636:28201100: maximum number of nested calls for this thread: 14. 1820.054576592:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 1820.054580503:28201100: 1820.054582458:28201100: Recorded Call Order for Thread 'action 1 queue:Reg/w0 (28201200)': 1820.054584414:28201100: 0: wtp.c:417:wtpWorker: 1820.054586649:28201100: 1: wti.c:403:wtiWorker: 1820.054588604:28201100: maximum number of nested calls for this thread: 4. 1820.054590560:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 1820.054592516:28201100: 1820.054594471:28201100: Recorded Call Order for Thread 'main queue:Reg/w0 (28201300)': 1820.054596427:28201100: 0: wtp.c:417:wtpWorker: 1820.054598662:28201100: 1: wti.c:403:wtiWorker: 1820.054600617:28201100: maximum number of nested calls for this thread: 4. 1820.054602573:28201100: NOTE: not all calls may have been recorded, code does not currently guarantee that! 1820.054606484:28201100: Mutex log for all known mutex operations: 1820.054608439:28201100: mutex 0x28215150 is being waited on by code at wti.c:347, thread action 1 queue:Reg/w0 (28201200) 1820.054610674:28201100: mutex 0x28215140 is being waited on by code at wti.c:347, thread main queue:Reg/w0 (28201300) 1820.054612630:28201100: If the call trace is empty, you may want to ./configure --enable-rtinst 1820.054614585:28201100: To submit bug reports, visit http://www.rsyslog.com/bugs 1820.057026344:28201100: To submit bug reports, visit http://www.rsyslog.com/bugs Abort (core dumped)

-HKS

by **rgerhards** » Fri Jul 11, 2008 5:15 pm

This is good information, it points consistently into the same direction. Could you also provide me a gdb backtrace (just like the last one). Together, they begin to make sense

I'll probably need to add some debug info and ask you to run that again. I am not sure if I manage this before monday...

Rainer

by **rgerhards** » Fri Jul 11, 2008 5:17 pm

hold a moment - I am doing a bit of instrumation right now

by **rgerhards** » Fri Jul 11, 2008 5:19 pm

Ok, it may not be the greatest debug info I added, but I think it is useful. Please do a new pull and re-try. A gdb backtrace after the abort would be great.

MonitorWare Knowledge Base

Core dump with non-Direct queue

Core dump with non-Direct queue

Professional Services Information

Learn more about rsyslog professional services.

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Re: Core dump with non-Direct queue

Google Ads

Who is online