MonitorWare Knowledge Base

Yet another weird one from yours truly.

rsyslogd seems to be forwarding messages with different priorities than they have when they are generated. I'm using this conf file:

Code: Select all

$ModLoad immark.so              # --MARK-- message capability
$ModLoad imuxsock.so            # support for local system logging
$ModLoad imklog.so              # kernel logging support
$ModLoad imudp.so               # UDP syslog input

$UDPServerAddress 10.123.0.101
$UDPServerRun 514

*.*                             @10.123.0.97:514;RSYSLOG_TraditionalForwardFormat

$template test, "%timegenerated% %fromhost% %syslogtag%%msg:::drop-last-lf% (%pri-text%: %syslogfacility% %syslogpriority%)\n"

*.notice;authpriv.info;kern.debug;lpr.info;mail.crit    /var/log/messages
auth.info;authpriv.info                                             /var/log/auth.log

*.*                                                                           /var/log/debug.log;test


A generic sudo command (sudo w) generates this message in /var/log/debug.log:

Code: Select all

Jul  7 13:40:08 qa-freebsd sudo:     root : TTY=ttyv1 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/w (local2.notice<149>: 18 5)


However, tcpdump on the machine's egress interface shows the following going out:

Code: Select all

13:40:08.168996 IP (tos 0x0, ttl 64, id 2912, offset 0, flags [none], proto UDP (17), length 130) qa-freebsd.53011 > test-destination.syslog: SYSLOG, length: 102
        Facility mail (2), Severity debug (7)


Here's a pair of tests using the logger commands:
logger -p local2.notice 'test local2.notice'
logger -p authpriv.notice 'test authpriv.notice'

And the results (order maintained):

Code: Select all

Syslog:
Jul  7 13:43:04 qa-freebsd root: test local2.notice (local2.notice<149>: 18 5)
Jul  7 13:43:05 qa-freebsd root: test authpriv.notice (authpriv.notice<85>: 10 5)

TCPDump:
13:43:04.822675 IP (tos 0x0, ttl 64, id 3518, offset 0, flags [none], proto UDP (17), length 83) qa-freebsd.54913 > test-destination.syslog: SYSLOG, length: 55
        Facility mail (2), Severity debug (7)
13:43:05.550521 IP (tos 0x0, ttl 64, id 3522, offset 0, flags [none], proto UDP (17), length 85) qa-freebsd.54913 > test-destination.syslog: SYSLOG, length: 57
        Facility user (1), Severity debug (7)


Is this something I have misconfigured, or some strangeness on the part of rsyslogd? I also find it interesting that sudo is logging with local2.notice rather than authpriv.notice (as it should be), but I don't know how to dig deeper into that.

-HKS

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

Nevermind on the sudo-logging-with-local2 vs authpriv difference - different version of sudo than I was used to.

-HKS

The difference between TCPDump and the properties is indeed strange. Could you run rsyslogd interactively in debug mode (add -dn) and provide me the debug log?

Also, I am not familiar with TCPDump (using Wireshark ). Is there any way that you can provide me the plain contents of the message you see on the wire? I mean without decoding it (the message is all printable text, so you do not need a decoder).

Thanks,
Rainer

Here's the full tcpdump output (no verbose options turned on):

Code: Select all

11:42:33.108196 IP qa-freebsd.57201 > test-destination.syslog: SYSLOG mail.debug, length: 102
E.......@.`.
{.e
{.a.q...n..<23>Jul  8 11:42:33 qa-freebsd sudo:     root : TTY=ttyv0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/w


The priority <23> does (by my calculations, at least) decode to mail.debug.

The -dn output has been emailed to you since it's too large for the forum

Thanks for the help.

-HKS

I agree, 23 is mail.debug. Will look at the log soon.

mhhh... the debug output does not tell me the PRI -- would it be possible that you update to 3.19.9, so that I can modify it with more elaborate debug? I do not like to do that to the beta...

Sure. Download the one off the main site, or do I need to download a special version?

-HKS

I'll prepare one for you, but that will probably be tomorrow morning. If you like, you can start with the one from the main site just to see if the issues disappears (I don't think so, but you never know ).

I'll wait till tomorrow. Thanks.

-HKS

The tarball is now available at

http://download.rsyslog.com/rsyslog/rsy ... st1.tar.gz

I ran a couple of test on it, and it works well in my lab. In any case, the debug now displays the priority of messages received. So we should get to a solution.

Rainer

Perhaps I'm missing something very simple, but in this version (and the vanilla 3.19.9), I'm unable to get rsyslog to forward logs at all. They never even hit the machine's interface.

I've reduced my config file to this:

Code: Select all

$ModLoad immark.so              # --MARK-- message capability
$ModLoad imuxsock.so            # support for local system logging
$ModLoad imklog.so              # kernel logging support


*.*                             @10.123.0.97:514
*.*                             /var/log/debug.log


Test log messages are deposited into /var/log/debug.log, but nowhere else. Debugging output:

Code: Select all

6680.385621747:main thread: Writing pidfile /var/run/rsyslogd.pid.
6680.387593226:main thread: rsyslog 3.19.10-test1 - called init()
6680.387595182:main thread: Unloading non-static modules.
6680.387597138:main thread: module lmnet NOT unloaded because it still has a refcount of 3
6680.387599373:main thread: Clearing templates.
6680.388131004:main thread: cfline: '$ModLoad imuxsock.so            # support for local system logging'
6680.388499487:main thread: Requested to load module 'imuxsock.so'
6680.388501722:main thread: loading module '/usr/local/lib/rsyslog/imuxsock.so'
6680.389526992:main thread: module of type 0 being loaded.
6680.389528947:main thread: cfline: '$ModLoad imklog.so              # kernel logging support'
6680.389530903:main thread: Requested to load module 'imklog.so'
6680.389533138:main thread: loading module '/usr/local/lib/rsyslog/imklog.so'
6680.390991424:main thread: module of type 0 being loaded.
6680.390993379:main thread: cfline: '*.*                             @10.123.0.97:514'
6680.390995335:main thread:  - traditional PRI filter
6680.390997290:main thread: symbolic name: * ==> 255
6680.391117417:main thread: tried selector action for builtin-file: -2001
6680.391119373:main thread: tried selector action for builtin-fwd: 0
6680.391123284:main thread: Module builtin-fwd processed this config line.
6680.391125519:main thread: template: 'RSYSLOG_TraditionalForwardFormat' assigned

6680.391588706:main thread: action 1 queue: save on shutdown 1, max disk space allowed 0
6680.391590941:main thread: action 1 queue: type 3, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting
6680.391592897:main thread: Action 0x2821f830: queue 0x28225140 created
6680.391594853:main thread: cfline: '*.*                                /var/log/debug.log'
6680.391596808:main thread: selector line successfully processed
6680.391598764:main thread:  - traditional PRI filter
6680.391600719:main thread: symbolic name: * ==> 255
6680.392530446:main thread: tried selector action for builtin-file: 0
6680.392997545:main thread: Module builtin-file processed this config line.
6680.392999500:main thread: template: 'RSYSLOG_FileFormat' assigned
6680.393001456:main thread: action 2 queue: save on shutdown 1, max disk space allowed 0
6680.393003411:main thread: action 2 queue: type 3, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting
6680.393005367:main thread: Action 0x2821f8d0: queue 0x28225380 created
6680.393151195:main thread: selector line successfully processed
6680.393568288:main thread: main queue: is NOT disk-assisted
6680.393570243:main thread: main queue: type 0, enq-only 0, disk assisted 0, maxFileSz 1048576, qsize 0, child 0 starting
6680.393572199:main thread: main queue:Reg: finalizing construction of worker thread pool
6680.393574154:main thread: main queue:Reg/w0: finalizing construction of worker instance data
6680.393576110:main thread: main queue: queue starts up without (loading) any DA disk state (this is normal for the DA queue itself!)
6680.393578065:main thread: main queue:Reg: high activity - starting 1 additional worker thread(s).
6680.393580300:main thread: main queue:Reg/w0: receiving command 2
6680.394628478:main thread: main queue:Reg: started with state 0, num workers now 1
6680.396992466:main thread: Main processing queue is initialized and running
6680.397559298:main thread: Opened UNIX socket '/var/run/log' (fd 3).
6680.398561939:main thread:
Active selectors:
6680.398604123:main thread: Selector 1:
6680.398606079:main thread: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Actions:
6680.400018828:main thread: builtin-fwd: 10.123.0.97
        Instance data: 0x28205680
6680.400074422:main thread:     RepeatedMsgReduction: 0
6680.400076657:main thread:     Resume Interval: 30
6680.400078612:main thread:     Suspended: 0
6680.400501292:main thread:     Disabled: 0
6680.400503247:main thread:     Exec only when previous is suspended: 0
6680.400505482:main thread:
6680.400507438:main thread:
6680.400509393:main thread:
6680.400511349:main thread: Selector 2:
6680.400513304:main thread: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Actions:
6680.401679654:main thread: builtin-file: /var/log/debug.log
        Instance data: 0x28225260
6680.401717647:main thread:     RepeatedMsgReduction: 0
6680.401719603:main thread:     Resume Interval: 30
6680.401721838:main thread:     Suspended: 0
6680.401750892:main thread:     Disabled: 0
6680.401753127:main thread:     Exec only when previous is suspended: 0
6680.401755082:main thread:
6680.401757038:main thread:
6680.401758993:main thread:
6680.401760949:main thread:
6680.401762905:main thread: Template: Name='RSYSLOG_DebugFormat'
6680.401780225:main thread:     Entry(2821c060): type 1, (CONSTANT), value: 'Debug line with all properties:
FROMHOST: ''
6680.401805647:main thread:     Entry(2821c0b0): type 2, (FIELD), value: 'fromhost'
6680.401827997:main thread:     Entry(2821c100): type 1, (CONSTANT), value: '', fromhost-ip: ''
6680.402253749:main thread:     Entry(2821c150): type 2, (FIELD), value: 'fromhost-ip'
6680.402308225:main thread:     Entry(2821c1a0): type 1, (CONSTANT), value: '', HOSTNAME: ''
6680.402362701:main thread:     Entry(2821c1f0): type 2, (FIELD), value: 'hostname'
6680.402416619:main thread:     Entry(2821c240): type 1, (CONSTANT), value: '', PRI: '
6680.402995184:main thread:     Entry(2821c290): type 2, (FIELD), value: 'pri'
6680.403051057:main thread:     Entry(2821c2e0): type 1, (CONSTANT), value: ',
syslogtag ''
6680.403126765:main thread:     Entry(2821c330): type 2, (FIELD), value: 'syslogtag'
6680.403501673:main thread:     Entry(2821c380): type 1, (CONSTANT), value: '', programname: ''
6680.403571794:main thread:     Entry(2821c3d0): type 2, (FIELD), value: 'programname'
6680.403625990:main thread:     Entry(2821c420): type 1, (CONSTANT), value: '', APP-NAME: ''
6680.404009000:main thread:     Entry(2821c470): type 2, (FIELD), value: 'app-name'
6680.404063476:main thread:     Entry(2821c4c0): type 1, (CONSTANT), value: '', PROCID: ''
6680.404117673:main thread:     Entry(2821c510): type 2, (FIELD), value: 'procid'
6680.404491743:main thread:     Entry(2821c560): type 1, (CONSTANT), value: '', MSGID: ''
6680.404561305:main thread:     Entry(2821c5b0): type 2, (FIELD), value: 'msgid'
6680.404615781:main thread:     Entry(2821c600): type 1, (CONSTANT), value: '',
TIMESTAMP: ''
6680.405017229:main thread:     Entry(2821c650): type 2, (FIELD), value: 'timestamp'
6680.405071705:main thread:     Entry(2821c6a0): type 1, (CONSTANT), value: '', STRUCTURED-DATA: ''
6680.405126181:main thread:     Entry(2821c6f0): type 2, (FIELD), value: 'structured-data'
6680.405503883:main thread:     Entry(2821c740): type 1, (CONSTANT), value: '',
msg: ''
6680.405584899:main thread:     Entry(2821c790): type 2, (FIELD), value: 'msg'
6680.405639095:main thread:     Entry(2821c7e0): type 1, (CONSTANT), value: ''
escaped msg: ''
6680.406049762:main thread:     Entry(2821c830): type 2, (FIELD), value: 'msg' [drop control-characters]
6680.406118207:main thread:     Entry(2821c880): type 1, (CONSTANT), value: ''
rawmsg: ''
6680.406491718:main thread:     Entry(2821c8d0): type 2, (FIELD), value: 'rawmsg'
6680.406574689:main thread:     Entry(2821c920): type 1, (CONSTANT), value: ''

'
6680.406651235:main thread: Template: Name='RSYSLOG_SyslogProtocol23Format'
6680.408059515:main thread:     Entry(2821c970): type 1, (CONSTANT), value: '<'
6680.408129077:main thread:     Entry(2821c9c0): type 2, (FIELD), value: 'pri'
6680.408502309:main thread:     Entry(2821ca10): type 1, (CONSTANT), value: '>1 '
6680.408572429:main thread:     Entry(2821ca60): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date]
6680.408640594:main thread:     Entry(2821cab0): type 1, (CONSTANT), value: ' '
6680.409038131:main thread:     Entry(2821cb00): type 2, (FIELD), value: 'hostname'
6680.409092607:main thread:     Entry(2821cb50): type 1, (CONSTANT), value: ' '
6680.409146804:main thread:     Entry(2821cba0): type 2, (FIELD), value: 'app-name'
6680.409535960:main thread:     Entry(2821cbf0): type 1, (CONSTANT), value: ' '
6680.409590436:main thread:     Entry(2821cc40): type 2, (FIELD), value: 'procid'
6680.409644633:main thread:     Entry(2821cc90): type 1, (CONSTANT), value: ' '
6680.410033229:main thread:     Entry(2821cce0): type 2, (FIELD), value: 'msgid'
6680.410087426:main thread:     Entry(2821cd30): type 1, (CONSTANT), value: ' '
6680.410141623:main thread:     Entry(2821cd80): type 2, (FIELD), value: 'structured-data'
6680.410145814:main thread:     Entry(2821cdd0): type 1, (CONSTANT), value: ' '
6680.410586931:main thread:     Entry(2821ce20): type 2, (FIELD), value: 'msg'
6680.410641128:main thread:     Entry(2821ce70): type 1, (CONSTANT), value: '
'
6680.411033357:main thread: Template: Name='RSYSLOG_FileFormat'
6680.411073306:main thread:     Entry(2821cec0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date]
6680.411141191:main thread:     Entry(2821cf10): type 1, (CONSTANT), value: ' '
6680.411145382:main thread:     Entry(2821cf60): type 2, (FIELD), value: 'hostname'
6680.411584544:main thread:     Entry(2821cfb0): type 1, (CONSTANT), value: ' '
6680.411651033:main thread:     Entry(2821e060): type 2, (FIELD), value: 'syslogtag'
6680.412034042:main thread:     Entry(2821e0b0): type 2, (FIELD), value: 'msg' [drop last LF in msg]
6680.412102487:main thread:     Entry(2821e100): type 1, (CONSTANT), value: '
'
6680.413616366:main thread: Template: Name='RSYSLOG_TraditionalFileFormat'
6680.413658550:main thread:     Entry(2821e150): type 2, (FIELD), value: 'timestamp'
6680.414044354:main thread:     Entry(2821e1a0): type 1, (CONSTANT), value: ' '
6680.414099389:main thread:     Entry(2821e1f0): type 2, (FIELD), value: 'hostname'
6680.414153306:main thread:     Entry(2821e240): type 1, (CONSTANT), value: ' '
6680.414561179:main thread:     Entry(2821e290): type 2, (FIELD), value: 'syslogtag'
6680.414615655:main thread:     Entry(2821e2e0): type 2, (FIELD), value: 'msg' [drop last LF in msg]
6680.415018779:main thread:     Entry(2821e330): type 1, (CONSTANT), value: '
'
6680.415084430:main thread: Template: Name=' WallFmt'
6680.415124100:main thread:     Entry(2821e380): type 1, (CONSTANT), value: '
Message from syslogd@'
6680.415499008:main thread:     Entry(2821e3d0): type 2, (FIELD), value: 'hostname'
6680.415582259:main thread:     Entry(2821e420): type 1, (CONSTANT), value: ' at '
6680.415636456:main thread:     Entry(2821e470): type 2, (FIELD), value: 'timegenerated'
6680.416020862:main thread:     Entry(2821e4c0): type 1, (CONSTANT), value: ' ...
'
6680.416087071:main thread:     Entry(2821e510): type 2, (FIELD), value: 'syslogtag'
6680.416140989:main thread:     Entry(2821e560): type 2, (FIELD), value: 'msg'
6680.416145179:main thread:     Entry(2821e5b0): type 1, (CONSTANT), value: '
'
6680.416596354:main thread: Template: Name='RSYSLOG_ForwardFormat'
6680.416636303:main thread:     Entry(2821e600): type 1, (CONSTANT), value: '<'
6680.417018754:main thread:     Entry(2821e650): type 2, (FIELD), value: 'pri'
6680.417073510:main thread:     Entry(2821e6a0): type 1, (CONSTANT), value: '>'
6680.417127148:main thread:     Entry(2821e6f0): type 2, (FIELD), value: 'timestamp' [Format as RFC3339-Date]
6680.417601789:main thread:     Entry(2821e740): type 1, (CONSTANT), value: ' '
6680.417657103:main thread:     Entry(2821e790): type 2, (FIELD), value: 'hostname'
6680.418033688:main thread:     Entry(2821e7e0): type 1, (CONSTANT), value: ' '
6680.419131592:main thread:     Entry(2821e830): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32]
6680.419503986:main thread:     Entry(2821e880): type 2, (FIELD), value: 'msg'
6680.419574107:main thread: Template: Name='RSYSLOG_TraditionalForwardFormat'
6680.419614056:main thread:     Entry(2821e8d0): type 1, (CONSTANT), value: '<'
6680.419995110:main thread:     Entry(2821e920): type 2, (FIELD), value: 'pri'
6680.420049586:main thread:     Entry(2821e970): type 1, (CONSTANT), value: '>'
6680.420125015:main thread:     Entry(2821e9c0): type 2, (FIELD), value: 'timestamp'
6680.420499644:main thread:     Entry(2821ea10): type 1, (CONSTANT), value: ' '
6680.420568926:main thread:     Entry(2821ea60): type 2, (FIELD), value: 'hostname'
6680.420622564:main thread:     Entry(2821eab0): type 1, (CONSTANT), value: ' '
6680.421066196:main thread:     Entry(2821eb00): type 2, (FIELD), value: 'syslogtag' [substring, from character 1 to 32]
6680.421139948:main thread:     Entry(2821eb50): type 2, (FIELD), value: 'msg'
6680.421143859:main thread: Template: Name=' StdUsrMsgFmt'
6680.421570729:main thread:     Entry(2821eba0): type 1, (CONSTANT), value: ' '
6680.421625206:main thread:     Entry(2821ebf0): type 2, (FIELD), value: 'syslogtag'
6680.422007936:main thread:     Entry(2821ec40): type 2, (FIELD), value: 'msg'
6680.422062691:main thread:     Entry(2821ec90): type 1, (CONSTANT), value: '
'
6680.422128621:main thread: Template: Name=' StdDBFmt' [SQL-Format (MySQL)]
6680.422502133:main thread:     Entry(2821ece0): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (''
6680.422572812:main thread:     Entry(2821ed30): type 2, (FIELD), value: 'msg'
6680.422627009:main thread:     Entry(2821ed80): type 1, (CONSTANT), value: '', '
6680.423010577:main thread:     Entry(2821edd0): type 2, (FIELD), value: 'syslogfacility'
6680.423079021:main thread:     Entry(2821ee20): type 1, (CONSTANT), value: ', ''
6680.423133498:main thread:     Entry(2821ee70): type 2, (FIELD), value: 'hostname'
6680.424547923:main thread:     Entry(2821eec0): type 1, (CONSTANT), value: '', '
6680.424603238:main thread:     Entry(2821ef10): type 2, (FIELD), value: 'syslogpriority'
6680.424657434:main thread:     Entry(2821ef60): type 1, (CONSTANT), value: ', ''
6680.425041282:main thread:     Entry(2821efb0): type 2, (FIELD), value: 'timereported' [Format as MySQL-Date]
6680.425109168:main thread:     Entry(2821f060): type 1, (CONSTANT), value: '', ''
6680.425498882:main thread:     Entry(2821f0b0): type 2, (FIELD), value: 'timegenerated' [Format as MySQL-Date]
6680.425568723:main thread:     Entry(2821f100): type 1, (CONSTANT), value: '', '
6680.425622920:main thread:     Entry(2821f150): type 2, (FIELD), value: 'iut'
6680.426027720:main thread:     Entry(2821f1a0): type 1, (CONSTANT), value: ', ''
6680.426082755:main thread:     Entry(2821f1f0): type 2, (FIELD), value: 'syslogtag'
6680.426137231:main thread:     Entry(2821f240): type 1, (CONSTANT), value: '')'
6680.426556838:main thread: Template: Name=' StdPgSQLFmt' [SQL-Format (standard SQL)]
6680.426611035:main thread:     Entry(2821f290): type 1, (CONSTANT), value: 'insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (''
6680.426999911:main thread:     Entry(2821f2e0): type 2, (FIELD), value: 'msg'
6680.427054387:main thread:     Entry(2821f330): type 1, (CONSTANT), value: '', '
6680.427108584:main thread:     Entry(2821f380): type 2, (FIELD), value: 'syslogfacility'
6680.427492711:main thread:     Entry(2821f3d0): type 1, (CONSTANT), value: ', ''
6680.427562273:main thread:     Entry(2821f420): type 2, (FIELD), value: 'hostname'
6680.427616470:main thread:     Entry(2821f470): type 1, (CONSTANT), value: '', '
6680.428005625:main thread:     Entry(2821f4c0): type 2, (FIELD), value: 'syslogpriority'
6680.428060102:main thread:     Entry(2821f510): type 1, (CONSTANT), value: ', ''
6680.428114019:main thread:     Entry(2821f560): type 2, (FIELD), value: 'timereported' [Format as PgSQL-Date]
6680.429572864:main thread:     Entry(2821f5b0): type 1, (CONSTANT), value: '', ''
6680.429641867:main thread:     Entry(2821f600): type 2, (FIELD), value: 'timegenerated' [Format as PgSQL-Date]
6680.430044991:main thread:     Entry(2821f650): type 1, (CONSTANT), value: '', '
6680.430100026:main thread:     Entry(2821f6a0): type 2, (FIELD), value: 'iut'
6680.430155060:main thread:     Entry(2821f6f0): type 1, (CONSTANT), value: ', ''
6680.430539746:main thread:     Entry(2821f740): type 2, (FIELD), value: 'syslogtag'
6680.430594781:main thread:     Entry(2821f790): type 1, (CONSTANT), value: '')'
6680.430650375:main thread: Loaded Module: Name='lmnet', IFVersion=4, type=library module.
6680.431051543:main thread: Entry points:
6680.431053499:main thread:     queryEtryPt:        0x281e44d0
6680.431055454:main thread:     doAction:           0x0
6680.431057410:main thread:     parseSelectorAct:   0x0
6680.431059365:main thread:     dbgPrintInstInfo:   0x0
6680.431063556:main thread:     freeInstance:       0x0
6680.431065511:main thread:
6680.431067467:main thread: Loaded Module: Name='builtin-file', IFVersion=4, type=output module.
6680.431993283:main thread: Entry points:
6680.431995518:main thread:     queryEtryPt:        0x80564c0
6680.431997474:main thread:     doAction:           0x80573f0
6680.431999429:main thread:     parseSelectorAct:   0x8056a30
6680.432001385:main thread:     dbgPrintInstInfo:   0x8057220
6680.432003340:main thread:     freeInstance:       0x80569b0
6680.432005296:main thread:
6680.432009486:main thread: Loaded Module: Name='builtin-fwd', IFVersion=4, type=output module.
6680.432582185:main thread: Entry points:
6680.432584140:main thread:     queryEtryPt:        0x8054d30
6680.432586096:main thread:     doAction:           0x8055ce0
6680.432588051:main thread:     parseSelectorAct:   0x8055220
6680.432590007:main thread:     dbgPrintInstInfo:   0x8055200
6680.432592242:main thread:     freeInstance:       0x8055100
6680.432594197:main thread:
6680.432596153:main thread: Loaded Module: Name='builtin-shell', IFVersion=4, type=output module.
6680.434938350:main thread: Entry points:
6680.434940306:main thread:     queryEtryPt:        0x8053ce0
6680.434942261:main thread:     doAction:           0x8054040
6680.434944217:main thread:     parseSelectorAct:   0x8053e70
6680.434946172:main thread:     dbgPrintInstInfo:   0x8054020
6680.434948407:main thread:     freeInstance:       0x8053e50
6680.434950363:main thread:
6680.434952318:main thread: Loaded Module: Name='builtin-discard', IFVersion=4, type=output module.
6680.434952318:main thread: Entry points:
6680.434954274:main thread:     queryEtryPt:        0x8057ab0
6680.434956229:main thread:     doAction:           0x8057c20
6680.434958185:main thread:     parseSelectorAct:   0x8057c60
6680.434960420:main thread:     dbgPrintInstInfo:   0x8057980
6680.434962376:main thread:     freeInstance:       0x8057c40
6680.434964331:main thread:
6680.434966287:main thread: Loaded Module: Name='builtin-usrmsg', IFVersion=4, type=output module.
6680.434966287:main thread: Entry points:
6680.434968242:main thread:     queryEtryPt:        0x8054200
6680.434970198:main thread:     doAction:           0x8054780
6680.434972433:main thread:     parseSelectorAct:   0x80543e0
6680.434974388:main thread:     dbgPrintInstInfo:   0x8054370
6680.434976344:main thread:     freeInstance:       0x80543c0
6680.434978299:main thread:
6680.434980255:main thread: Loaded Module: Name='imuxsock.so', IFVersion=4, type=input module.
6680.435668331:main thread: Entry points:
6680.435670566:main thread:     queryEtryPt:        0x281e9ec0
6680.435672522:main thread:     doAction:           0x281ea3d0
6680.435674477:main thread:     parseSelectorAct:   0x281ea190
6680.435676433:main thread:     dbgPrintInstInfo:   0x0
6680.435678388:main thread:     freeInstance:       0x0
6680.435680344:main thread:
6680.435682579:main thread: Loaded Module: Name='imklog.so', IFVersion=4, type=input module.
6680.436503353:main thread: Entry points:
6680.436505309:main thread:     queryEtryPt:        0x281ed290
6680.436507265:main thread:     doAction:           0x281ed410
6680.438010808:main thread:     parseSelectorAct:   0x281ed3f0
6680.438091823:28201200: main queue:Reg/w0: receiving command 4
6680.440582084:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3
6680.441021246:imklog.c: ----------imklog waiting for kernel log line
6680.441077119:main thread:     dbgPrintInstInfo:   0x0
6680.441079074:main thread:     freeInstance:       0x0
6680.441081030:main thread:
6680.441084941:main thread: Sytem Line Configuration Commands:
6680.441086897:main thread:     Command 'workdirectory':
6680.441089132:main thread:             type : 11
6680.441091087:main thread:             pData: 0x8091ac4
6680.441093043:main thread:             Hdlr : 0x0
6680.441094998:main thread:             Owner: 0x0
6680.441096954:main thread:
6680.441098910:main thread:     Command 'dropmsgswithmaliciousdnsptrrecords':
6680.441101144:main thread:             type : 4
6680.441103100:main thread:             pData: 0x8091ad0
6680.441105056:main thread:             Hdlr : 0x0
6680.441107011:main thread:             Owner: 0x0
6680.442575633:main thread:
6680.442577589:main thread:     Command 'defaultnetstreamdriver':
6680.442579824:main thread:             type : 11
6680.442581780:main thread:             pData: 0x8091ab4
6680.442583735:main thread:             Hdlr : 0x0
6680.442585691:main thread:             Owner: 0x0
6680.442587646:main thread:
6680.442589602:main thread:     Command 'defaultnetstreamdrivercafile':
6680.442591837:main thread:             type : 11
6680.442593792:main thread:             pData: 0x8091ab8
6680.442597703:main thread:             Hdlr : 0x0
6680.442599659:main thread:             Owner: 0x0
6680.442601614:main thread:
6680.442603849:main thread:     Command 'defaultnetstreamdriverkeyfile':
6680.442605805:main thread:             type : 11
6680.442607760:main thread:             pData: 0x8091abc
6680.442609716:main thread:             Hdlr : 0x0
6680.442611672:main thread:             Owner: 0x0
6680.442613627:main thread:
6680.442615862:main thread:     Command 'defaultnetstreamdrivercertfile':
6680.442617818:main thread:             type : 11
6680.442619773:main thread:             pData: 0x8091ac0
6680.442621729:main thread:             Hdlr : 0x0
6680.442625640:main thread:             Owner: 0x0
6680.442627595:main thread:
6680.442629830:main thread:     Command 'resetconfigvariables':
6680.445009742:main queue:Reg/w0: main queue: entering rate limiter
6680.447139621:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work.
6680.447147723:main thread:             type : 1
6680.447149679:main thread:             pData: 0x0
6680.447151634:main thread:             Hdlr : 0x8058bd0
6680.447153590:main thread:             Owner: 0x0
6680.447155825:main thread:
6680.447157780:main thread:             type : 1
6680.447159736:main thread:             pData: 0x0
6680.447161691:main thread:             Hdlr : 0x80568a0
6680.447163647:main thread:             Owner: 0x8056880
6680.447165602:main thread:
6680.447169793:main thread:             type : 1
6680.447171748:main thread:             pData: 0x0
6680.447173704:main thread:             Hdlr : 0x8054fa0
6680.447175659:main thread:             Owner: 0x8054fd0
6680.447177615:main thread:
6680.447179571:main thread:             type : 1
6680.447181805:main thread:             pData: 0x0
6680.447183761:main thread:             Hdlr : 0x804ee10
6680.447185717:main thread:             Owner: 0x0
6680.447187672:main thread:
6680.447189628:main thread:             type : 1
6680.447191583:main thread:             pData: 0x0
6680.447193818:main thread:             Hdlr : 0x281ea070
6680.447197729:main thread:             Owner: 0x281e9b40
6680.447199685:main thread:
6680.447201641:main thread:             type : 1
6680.447203596:main thread:             pData: 0x0
6680.447205552:main thread:             Hdlr : 0x281ed220
6680.447207787:main thread:             Owner: 0x281ece80
6680.447209742:main thread:
6680.447211698:main thread:     Command 'dynafilecachesize':
6680.447213653:main thread:             type : 6
6680.447215609:main thread:             pData: 0x0
6680.447217564:main thread:             Hdlr : 0x8056630
6680.447219799:main thread:             Owner: 0x8056880
6680.447223710:main thread:
6680.447225666:main thread:     Command 'dirowner':
6680.447227621:main thread:             type : 2
6680.447229577:main thread:             pData: 0x8091a84
6680.447231812:main thread:             Hdlr : 0x0
6680.447233767:main thread:             Owner: 0x8056880
6680.447235723:main thread:
6680.447237679:main thread:     Command 'dirgroup':
6680.447239634:main thread:             type : 3
6680.447241590:main thread:             pData: 0x8091a88
6680.447243825:main thread:             Hdlr : 0x0
6680.447245780:main thread:             Owner: 0x8056880
6680.451555266:main thread:
6680.451557222:main thread:     Command 'fileowner':
6680.451559178:main thread:             type : 2
6680.451561133:main thread:             pData: 0x8091a7c
6680.451563089:main thread:             Hdlr : 0x0
6680.451565324:main thread:             Owner: 0x8056880
6680.451567279:main thread:
6680.451569235:main thread:     Command 'filegroup':
6680.451571190:main thread:             type : 3
6680.451573146:main thread:             pData: 0x8091a80
6680.451575101:main thread:             Hdlr : 0x0
6680.451579292:main thread:             Owner: 0x8056880
6680.451581247:main thread:
6680.451583203:main thread:     Command 'dircreatemode':
6680.451585158:main thread:             type : 5
6680.451587114:main thread:             pData: 0x808f3d0
6680.451589070:main thread:             Hdlr : 0x0
6680.451591304:main thread:             Owner: 0x8056880
6680.451593260:main thread:
6680.451595216:main thread:     Command 'filecreatemode':
6680.451597171:main thread:             type : 5
6680.451599127:main thread:             pData: 0x808f3d4
6680.451601082:main thread:             Hdlr : 0x0
6680.451605273:main thread:             Owner: 0x8056880
6680.451607228:main thread:
6680.451609184:main thread:     Command 'createdirs':
6680.451611139:main thread:             type : 4
6680.451613095:main thread:             pData: 0x8091a8c
6680.451615330:main thread:             Hdlr : 0x0
6680.451617285:main thread:             Owner: 0x8056880
6680.451619241:main thread:
6680.451621197:main thread:     Command 'failonchownfailure':
6680.451623152:main thread:             type : 4
6680.451625108:main thread:             pData: 0x8091a78
6680.451627343:main thread:             Hdlr : 0x0
6680.451629298:main thread:             Owner: 0x8056880
6680.451633209:main thread:
6680.451635165:main thread:     Command 'actionfileenablesync':
6680.451637120:main thread:             type : 4
6680.451639076:main thread:             pData: 0x8091a6c
6680.451641311:main thread:             Hdlr : 0x0
6680.451643266:main thread:             Owner: 0x8056880
6680.451645222:main thread:
6680.451647178:main thread:     Command 'actionfiledefaulttemplate':
6680.451649133:main thread:             type : 11
6680.451651089:main thread:             pData: 0x8091a70
6680.451653324:main thread:             Hdlr : 0x0
6680.455642657:main thread:             Owner: 0x0
6680.455644613:main thread:
6680.455646568:main thread:     Command 'actionforwarddefaulttemplate':
6680.455648524:main thread:             type : 11
6680.455650480:main thread:             pData: 0x8091a50
6680.455652435:main thread:             Hdlr : 0x0
6680.455654670:main thread:             Owner: 0x0
6680.455656626:main thread:
6680.455658581:main thread:     Command 'actionsendstreamdriver':
6680.455662492:main thread:             type : 11
6680.455664448:main thread:             pData: 0x8091a54
6680.455666683:main thread:             Hdlr : 0x0
6680.455668638:main thread:             Owner: 0x0
6680.455670594:main thread:
6680.455672550:main thread:     Command 'actionsendstreamdrivermode':
6680.455674505:main thread:             type : 6
6680.455676461:main thread:             pData: 0x8091a58
6680.455678696:main thread:             Hdlr : 0x0
6680.455680651:main thread:             Owner: 0x0
6680.455682607:main thread:
6680.455684562:main thread:     Command 'actionsendstreamdriverauthmode':
6680.455686518:main thread:             type : 11
6680.457526416:main thread:             pData: 0x8091a5c
6680.457528372:main thread:             Hdlr : 0x0
6680.457530607:main thread:             Owner: 0x0
6680.457532563:main thread:
6680.457534518:main thread:     Command 'actionsendstreamdriverpermittedpeer':
6680.457536474:main thread:             type : 11
6680.457538429:main thread:             pData: 0x0
6680.457540385:main thread:             Hdlr : 0x80550c0
6680.457542620:main thread:             Owner: 0x0
6680.457544575:main thread:
6680.457546531:main thread:     Command 'actionsendresendlastmsgonreconnect':
6680.457548486:main thread:             type : 4
6680.457552397:main thread:             pData: 0x8091a60
6680.457554632:main thread:             Hdlr : 0x0
6680.457556588:main thread:             Owner: 0x0
6680.457558543:main thread:
6680.457560499:main thread:     Command 'actionresumeretrycount':
6680.457562454:main thread:             type : 6
6680.457564410:main thread:             pData: 0x8092a98
6680.457566366:main thread:             Hdlr : 0x0
6680.457568601:main thread:             Owner: 0x0
6680.457570556:main thread:
6680.457572512:main thread:     Command 'mainmsgqueuefilename':
6680.460144626:main thread:             type : 11
6680.460146582:main thread:             pData: 0x80919b4
6680.460148537:main thread:             Hdlr : 0x0
6680.460152728:main thread:             Owner: 0x0
6680.460154683:main thread:
6680.460156639:main thread:     Command 'mainmsgqueuesize':
6680.460158594:main thread:             type : 6
6680.460160550:main thread:             pData: 0x808ea40
6680.460162505:main thread:             Hdlr : 0x0
6680.460164461:main thread:             Owner: 0x0
6680.460166696:main thread:
6680.460168652:main thread:     Command 'mainmsgqueuehighwatermark':
6680.460170607:main thread:             type : 6
6680.460172563:main thread:             pData: 0x808ea44
6680.460174518:main thread:             Hdlr : 0x0
6680.460178709:main thread:             Owner: 0x0
6680.460180664:main thread:
6680.460182620:main thread:     Command 'mainmsgqueuelowwatermark':
6680.460184575:main thread:             type : 6
6680.460186531:main thread:             pData: 0x808ea48
6680.460188487:main thread:             Hdlr : 0x0
6680.460190722:main thread:             Owner: 0x0
6680.460192677:main thread:
6680.460194633:main thread:     Command 'mainmsgqueuediscardmark':
6680.460196588:main thread:             type : 6
6680.460198544:main thread:             pData: 0x808ea4c
6680.460200499:main thread:             Hdlr : 0x0
6680.460202455:main thread:             Owner: 0x0
6680.460206645:main thread:
6680.460208601:main thread:     Command 'mainmsgqueuediscardseverity':
6680.460210556:main thread:             type : 10
6680.460212512:main thread:             pData: 0x808ea50
6680.460214467:main thread:             Hdlr : 0x0
6680.460216702:main thread:             Owner: 0x0
6680.463072093:main thread:
6680.463074049:main thread:     Command 'mainmsgqueuecheckpointinterval':
6680.463076284:main thread:             type : 6
6680.463078239:main thread:             pData: 0x80919b8
6680.463082150:main thread:             Hdlr : 0x0
6680.463084106:main thread:             Owner: 0x0
6680.463086062:main thread:
6680.463088297:main thread:     Command 'mainmsgqueuetype':
6680.463090252:main thread:             type : 11
6680.463092208:main thread:             pData: 0x0
6680.463094163:main thread:             Hdlr : 0x804f480
6680.463096119:main thread:             Owner: 0x0
6680.463098074:main thread:
6680.465103357:main thread:     Command 'mainmsgqueueworkerthreads':
6680.465105313:main thread:             type : 6
6680.465107268:main thread:             pData: 0x808ea54
6680.465111179:main thread:             Hdlr : 0x0
6680.465113135:main thread:             Owner: 0x0
6680.465115370:main thread:
6680.465117325:main thread:     Command 'mainmsgqueuetimeoutshutdown':
6680.465119281:main thread:             type : 6
6680.465121236:main thread:             pData: 0x80919bc
6680.465123192:main thread:             Hdlr : 0x0
6680.465125147:main thread:             Owner: 0x0
6680.465127103:main thread:
6680.465129338:main thread:     Command 'mainmsgqueuetimeoutactioncompletion':
6680.465131293:main thread:             type : 6
6680.465133249:main thread:             pData: 0x808ea58
6680.465135205:main thread:             Hdlr : 0x0
6680.465139116:main thread:             Owner: 0x0
6680.465141351:main thread:
6680.465143306:main thread:     Command 'mainmsgqueuetimeoutenqueue':
6680.465145262:main thread:             type : 6
6680.465147217:main thread:             pData: 0x808ea5c
6680.465149173:main thread:             Hdlr : 0x0
6680.465151129:main thread:             Owner: 0x0
6680.465153363:main thread:
6680.465155319:main thread:     Command 'mainmsgqueueworketimeoutrthreadshutdown':
6680.465157275:main thread:             type : 6
6680.465159230:main thread:             pData: 0x808ea60
6680.465161186:main thread:             Hdlr : 0x0
6680.467527967:main thread:             Owner: 0x0
6680.467529922:main thread:
6680.467531878:main thread:     Command 'mainmsgqueuedequeueslowdown':
6680.467534113:main thread:             type : 6
6680.467536068:main thread:             pData: 0x80919c0
6680.467538024:main thread:             Hdlr : 0x0
6680.467539980:main thread:             Owner: 0x0
6680.467541935:main thread:
6680.467543891:main thread:     Command 'mainmsgqueueworkerthreadminimummessages':
6680.467545846:main thread:             type : 6
6680.467548081:main thread:             pData: 0x808ea64
6680.467550037:main thread:             Hdlr : 0x0
6680.467553948:main thread:             Owner: 0x0
6680.467555903:main thread:
6680.467557859:main thread:     Command 'mainmsgqueuemaxfilesize':
6680.467560094:main thread:             type : 7
6680.467562049:main thread:             pData: 0x808ea68
6680.469665389:main thread:             Hdlr : 0x0
6680.469667345:main thread:             Owner: 0x0
6680.469669301:main thread:
6680.469671256:main thread:     Command 'mainmsgqueuemaxdiskspace':
6680.469673212:main thread:             type : 7
6680.469675447:main thread:             pData: 0x80919c8
6680.469677402:main thread:             Hdlr : 0x0
6680.469679358:main thread:             Owner: 0x0
6680.469683269:main thread:
6680.469685224:main thread:     Command 'mainmsgqueuesaveonshutdown':
6680.469687459:main thread:             type : 4
6680.469689415:main thread:             pData: 0x808ea70
6680.469691370:main thread:             Hdlr : 0x0
6680.469693326:main thread:             Owner: 0x0
6680.469695281:main thread:
6680.469697237:main thread:     Command 'mainmsgqueuedequeuetimebegin':
6680.469699472:main thread:             type : 6
6680.469701427:main thread:             pData: 0x80919d0
6680.469703383:main thread:             Hdlr : 0x0
6680.469705339:main thread:             Owner: 0x0
6680.469707294:main thread:
6680.469711205:main thread:     Command 'mainmsgqueuedequeuetimeend':
6680.469713440:main thread:             type : 6
6680.469715396:main thread:             pData: 0x808ea74
6680.469717351:main thread:             Hdlr : 0x0
6680.469719307:main thread:             Owner: 0x0
6680.469721262:main thread:
6680.469723218:main thread:     Command 'repeatedmsgreduction':
6680.469725453:main thread:             type : 4
6680.469727408:main thread:             pData: 0x8092af4
6680.469729364:main thread:             Hdlr : 0x0
6680.469731319:main thread:             Owner: 0x0
6680.469733275:main thread:
6680.469737465:main thread:     Command 'actionexeconlywhenpreviousissuspended':
6680.469739421:main thread:             type : 4
6680.469741377:main thread:             pData: 0x8092af8
6680.469743332:main thread:             Hdlr : 0x0
6680.469745288:main thread:             Owner: 0x0
6680.469747243:main thread:
6680.469749478:main thread:     Command 'actionexeconlyonceeveryinterval':
6680.469751434:main thread:             type : 6
6680.469753389:main thread:             pData: 0x809199c
6680.469755345:main thread:             Hdlr : 0x0
6680.469757301:main thread:             Owner: 0x0
6680.469759256:main thread:
6680.469761212:main thread:     Command 'actionresumeinterval':
6680.474508742:main thread:             type : 6
6680.474510977:main thread:             pData: 0x0
6680.474512933:main thread:             Hdlr : 0x804f470
6680.474514888:main thread:             Owner: 0x0
6680.474516844:main thread:
6680.474518799:main thread:     Command 'controlcharacterescapeprefix':
6680.474520755:main thread:             type : 8
6680.474522711:main thread:             pData: 0x808ea78
6680.474524945:main thread:             Hdlr : 0x0
6680.474526901:main thread:             Owner: 0x0
6680.474528857:main thread:
6680.474530812:main thread:     Command 'escapecontrolcharactersonreceive':
6680.474532768:main thread:             type : 4
6680.474534723:main thread:             pData: 0x808ea7c
6680.474536958:main thread:             Hdlr : 0x0
6680.474538914:main thread:             Owner: 0x0
6680.474542825:main thread:
6680.474544781:main thread:     Command 'droptrailinglfonreception':
6680.474546736:main thread:             type : 4
6680.474548971:main thread:             pData: 0x808ea80
6680.474550927:main thread:             Hdlr : 0x0
6680.474552882:main thread:             Owner: 0x0
6680.474554838:main thread:
6680.474556793:main thread:     Command 'template':
6680.474558749:main thread:             type : 1
6680.474560704:main thread:             pData: 0x0
6680.474562939:main thread:             Hdlr : 0x805a750
6680.474564895:main thread:             Owner: 0x0
6680.474568806:main thread:
6680.474570761:main thread:     Command 'outchannel':
6680.474572717:main thread:             type : 1
6680.474574952:main thread:             pData: 0x1
6680.474576907:main thread:             Hdlr : 0x805a750
6680.474578863:main thread:             Owner: 0x0
6680.474580819:main thread:
6680.474582774:main thread:     Command 'allowedsender':
6680.474584730:main thread:             type : 1
6680.474586965:main thread:             pData: 0x2
6680.474588920:main thread:             Hdlr : 0x805a750
6680.474590876:main thread:             Owner: 0x0
6680.474592831:main thread:
6680.477549073:main thread:     Command 'modload':
6680.477551028:main thread:             type : 1
6680.477552984:main thread:             pData: 0x0
6680.477554940:main thread:             Hdlr : 0x805a850
6680.477556895:main thread:             Owner: 0x0
6680.477558851:main thread:
6680.479027752:main thread:     Command 'includeconfig':
6680.479029708:main thread:             type : 1
6680.479031943:main thread:             pData: 0x0
6680.479033899:main thread:             Hdlr : 0x805a920
6680.479035854:main thread:             Owner: 0x0
6680.479039765:main thread:
6680.479041721:main thread:     Command 'umask':
6680.479043956:main thread:             type : 5
6680.479045911:main thread:             pData: 0x0
6680.479047867:main thread:             Hdlr : 0x804f440
6680.479049822:main thread:             Owner: 0x0
6680.479051778:main thread:
6680.479053733:main thread:     Command 'debugprinttemplatelist':
6680.479055689:main thread:             type : 4
6680.479057924:main thread:             pData: 0x808ea84
6680.479059879:main thread:             Hdlr : 0x0
6680.479061835:main thread:             Owner: 0x0
6680.479065746:main thread:
6680.479067702:main thread:     Command 'debugprintmodulelist':
6680.479069937:main thread:             type : 4
6680.479071892:main thread:             pData: 0x808ea88
6680.479073848:main thread:             Hdlr : 0x0
6680.479075803:main thread:             Owner: 0x0
6680.479077759:main thread:
6680.479079714:main thread:     Command 'debugprintcfsyslinehandlerlist':
6680.479081949:main thread:             type : 4
6680.479083905:main thread:             pData: 0x808ea8c
6680.479085860:main thread:             Hdlr : 0x0
6680.479087816:main thread:             Owner: 0x0
6680.481548464:main thread:
6680.481550419:main thread:     Command 'moddir':
6680.481552375:main thread:             type : 11
6680.481554331:main thread:             pData: 0x8092a50
6680.481556286:main thread:             Hdlr : 0x0
6680.481558521:main thread:             Owner: 0x0
6680.481560477:main thread:
6680.481562432:main thread:     Command 'actionqueuefilename':
6680.481564388:main thread:             type : 11
6680.481566343:main thread:             pData: 0x8092a9c
6680.481568299:main thread:             Hdlr : 0x0
6680.481572489:main thread:             Owner: 0x0
6680.481574445:main thread:
6680.481576400:main thread:     Command 'actionqueuesize':
6680.481578356:main thread:             type : 6
6680.481580311:main thread:             pData: 0x8090740
6680.481582546:main thread:             Hdlr : 0x0
6680.481584502:main thread:             Owner: 0x0
6680.484032299:main thread:
6680.484034255:main thread:     Command 'actionqueuemaxdiskspace':
6680.484036210:main thread:             type : 7
6680.484038166:main thread:             pData: 0x8092aa0
6680.484040121:main thread:             Hdlr : 0x0
6680.484044312:main thread:             Owner: 0x0
6680.484046267:main thread:
6680.484048223:main thread:     Command 'actionqueuehighwatermark':
6680.484050178:main thread:             type : 6
6680.484052134:main thread:             pData: 0x8090744
6680.484054089:main thread:             Hdlr : 0x0
6680.484056045:main thread:             Owner: 0x0
6680.484058280:main thread:
6680.484060236:main thread:     Command 'actionqueuelowwatermark':
6680.484062191:main thread:             type : 6
6680.484064147:main thread:             pData: 0x8090748
6680.484066102:main thread:             Hdlr : 0x0
6680.484068058:main thread:             Owner: 0x0
6680.484072248:main thread:
6680.484074204:main thread:     Command 'actionqueuediscardmark':
6680.484076160:main thread:             type : 6
6680.484078115:main thread:             pData: 0x809074c
6680.484080071:main thread:             Hdlr : 0x0
6680.484082306:main thread:             Owner: 0x0
6680.484084261:main thread:
6680.484086217:main thread:     Command 'actionqueuediscardseverity':
6680.484088172:main thread:             type : 6
6680.484090128:main thread:             pData: 0x8090750
6680.484092083:main thread:             Hdlr : 0x0
6680.484094318:main thread:             Owner: 0x0
6680.484098229:main thread:
6680.484100185:main thread:     Command 'actionqueuecheckpointinterval':
6680.484102140:main thread:             type : 6
6680.484104096:main thread:             pData: 0x8092aa8
6680.484106051:main thread:             Hdlr : 0x0
6680.484108286:main thread:             Owner: 0x0
6680.484110242:main thread:
6680.484112197:main thread:     Command 'actionqueuetype':
6680.484114153:main thread:             type : 11
6680.484116109:main thread:             pData: 0x0
6680.484118064:main thread:             Hdlr : 0x807cf70
6680.484120299:main thread:             Owner: 0x0
6680.484122255:main thread:
6680.487525760:main thread:     Command 'actionqueueworkerthreads':
6680.487527995:main thread:             type : 6
6680.487529950:main thread:             pData: 0x8090754
6680.488604668:main thread:             Hdlr : 0x0
6680.488606624:main thread:             Owner: 0x0
6680.488608579:main thread:
6680.488610814:main thread:     Command 'actionqueuetimeoutshutdown':
6680.488612770:main thread:             type : 6
6680.488614725:main thread:             pData: 0x8092aac
6680.488616681:main thread:             Hdlr : 0x0
6680.488618636:main thread:             Owner: 0x0
6680.488622547:main thread:
6680.488624782:main thread:     Command 'actionqueuetimeoutactioncompletion':
6680.488626738:main thread:             type : 6
6680.488628693:main thread:             pData: 0x8090758
6680.488630649:main thread:             Hdlr : 0x0
6680.488632604:main thread:             Owner: 0x0
6680.488634560:main thread:
6680.488636795:main thread:     Command 'actionqueuetimeoutenqueue':
6680.488638750:main thread:             type : 6
6680.488640706:main thread:             pData: 0x809075c
6680.488642662:main thread:             Hdlr : 0x0
6680.488644617:main thread:             Owner: 0x0
6680.488646573:main thread:
6680.488650763:main thread:     Command 'actionqueueworkertimeoutthreadshutdown':
6680.488652719:main thread:             type : 6
6680.488654674:main thread:             pData: 0x8090760
6680.488656630:main thread:             Hdlr : 0x0
6680.488658585:main thread:             Owner: 0x0
6680.488660820:main thread:
6680.488662776:main thread:     Command 'actionqueueworkerthreadminimummessages':
6680.488664731:main thread:             type : 6
6680.488666687:main thread:             pData: 0x8090764
6680.488668642:main thread:             Hdlr : 0x0
6680.488670598:main thread:             Owner: 0x0
6680.488672554:main thread:
6680.488674788:main thread:     Command 'actionqueuemaxfilesize':
6680.488678700:main thread:             type : 7
6680.488680655:main thread:             pData: 0x8090768
6680.488682611:main thread:             Hdlr : 0x0
6680.488684566:main thread:             Owner: 0x0
6680.488686801:main thread:
6680.488688757:main thread:     Command 'actionqueuesaveonshutdown':
6680.488690712:main thread:             type : 4
6680.488692668:main thread:             pData: 0x8090770
6680.488694624:main thread:             Hdlr : 0x0
6680.488696579:main thread:             Owner: 0x0
6680.488698814:main thread:
6680.488702725:main thread:     Command 'actionqueuedequeueslowdown':
6680.493505570:main thread:             type : 6
6680.493507526:main thread:             pData: 0x8092ab0
6680.493509761:main thread:             Hdlr : 0x0
6680.493511716:main thread:             Owner: 0x0
6680.493513672:main thread:
6680.493515627:main thread:     Command 'actionqueuedequeuetimebegin':
6680.493517583:main thread:             type : 6
6680.493519538:main thread:             pData: 0x8092ab4
6680.493521494:main thread:             Hdlr : 0x0
6680.493523729:main thread:             Owner: 0x0
6680.493525685:main thread:
6680.493527640:main thread:     Command 'actionqueuedequeuetimeend':
6680.494548720:main thread:             type : 6
6680.494550675:main thread:             pData: 0x8090774
6680.494552631:main thread:             Hdlr : 0x0
6680.494554586:main thread:             Owner: 0x0
6680.494556542:main thread:
6680.494558777:main thread:     Command 'omitlocallogging':
6680.494560732:main thread:             type : 4
6680.494562688:main thread:             pData: 0x281ebe80
6680.494564643:main thread:             Hdlr : 0x0
6680.494566599:main thread:             Owner: 0x281e9b40
6680.494568554:main thread:
6680.494572745:main thread:     Command 'inputunixlistensocketignoremsgtimestamp':
6680.494574700:main thread:             type : 4
6680.494576656:main thread:             pData: 0x281ebc50
6680.494578612:main thread:             Hdlr : 0x0
6680.494580567:main thread:             Owner: 0x281e9b40
6680.494582523:main thread:
6680.494584758:main thread:     Command 'systemlogsocketname':
6680.494586713:main thread:             type : 11
6680.494588669:main thread:             pData: 0x281ebe84
6680.494590624:main thread:             Hdlr : 0x0
6680.494592580:main thread:             Owner: 0x281e9b40
6680.494594535:main thread:
6680.494596770:main thread:     Command 'addunixlistensocket':
6680.494600681:main thread:             type : 11
6680.494602637:main thread:             pData: 0x0
6680.494604592:main thread:             Hdlr : 0x281e9ab0
6680.494606548:main thread:             Owner: 0x281e9b40
6680.494608783:main thread:
6680.494610738:main thread:     Command 'systemlogsocketignoremsgtimestamp':
6680.494612694:main thread:             type : 4
6680.494614650:main thread:             pData: 0x0
6680.494616605:main thread:             Hdlr : 0x281e9a80
6680.498534701:main thread:             Owner: 0x281e9b40
6680.498536657:main thread:
6680.498538612:main thread:     Command 'debugprintkernelsymbols':
6680.498540847:main thread:             type : 4
6680.498542803:main thread:             pData: 0x281eef58
6680.498544758:main thread:             Hdlr : 0x0
6680.498546714:main thread:             Owner: 0x281ece80
6680.498548669:main thread:
6680.498550625:main thread:     Command 'klogsymbollookup':
6680.498552860:main thread:             type : 4
6680.498554815:main thread:             pData: 0x281eef64
6680.498556771:main thread:             Hdlr : 0x0
6680.498560682:main thread:             Owner: 0x281ece80
6680.498562637:main thread:
6680.498564872:main thread:     Command 'klogsymbolstwice':
6680.498566828:main thread:             type : 4
6680.498568783:main thread:             pData: 0x281eef5c
6680.498570739:main thread:             Hdlr : 0x0
6680.498572695:main thread:             Owner: 0x281ece80
6680.498574650:main thread:
6680.498576606:main thread:     Command 'klogusesyscallinterface':
6680.498578841:main thread:             type : 4
6680.498580796:main thread:             pData: 0x281eef60
6680.498582752:main thread:             Hdlr : 0x0
6680.498586663:main thread:             Owner: 0x281ece80
6680.500650054:main thread:
6680.500652009:main thread:     Command 'klogpermitnonkernelfacility':
6680.500653965:main thread:             type : 4
6680.500655920:main thread:             pData: 0x281eef68
6680.500658155:main thread:             Hdlr : 0x0
6680.500660111:main thread:             Owner: 0x281ece80
6680.500662066:main thread:
6680.500665977:main thread:     Command 'kloginternalmsgfacility':
6680.500667933:main thread:             type : 9
6680.500669888:main thread:             pData: 0x281eef74
6680.500672123:main thread:             Hdlr : 0x0
6680.500674079:main thread:             Owner: 0x281ece80
6680.500676034:main thread:
6680.500677990:main thread:
6680.500679946:main thread: Messages with malicious PTR DNS Records are not dropped.
6680.500681901:main thread: Control characters are replaced upon reception.
6680.500684136:main thread: Control character escape sequence prefix is '#'.
6680.500686092:main thread: Main queue size 10000 messages.
6680.503148695:main thread: Main queue worker threads: 1, Perists every 0 updates.
6680.503152606:main thread: Main queue timeouts: shutdown: 0, action completion shutdown: 1000, enq: 2000
6680.503154562:main thread: Main queue watermarks: high: 8000, low: 2000, discard: 9800, discard-severity: 4
6680.503156517:main thread: Main queue save on shutdown 1, max disk space allowed 0
6680.503158473:main thread: Work Directory: ''.
6680.504505292:main thread: logmsg: flags 5, pri 7, from 'qa-freebsd', msg  [origin software="rsyslogd" swVersion="3.19.10-test1" x-pid="773" x-info="http://www.rsyslog.com"] restart
6680.504507248:main thread: Message has legacy syslog format.
6680.504515349:main thread: main queue: entry added, size now 1 entries
6680.507057572:main thread: main queue: EnqueueMsg signaled condition (0)
6680.508036188:main queue:Reg/w0: main queue: entering rate limiter
6680.508070270:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries
6680.508105470:main queue:Reg/w0: Called action, logging to builtin-fwd
6680.508613077:main queue:Reg/w0:  10.123.0.97
6680.509011731:main queue:Reg/w0: Action requested to be suspended, done that.
6680.509015921:main queue:Reg/w0: Called action, logging to builtin-file
6680.509019832:main queue:Reg/w0:  (/var/log/debug.log)
6680.509499782:main thread: wtpAdviseMaxWorkers signals busy
6680.509540010:main thread:  (re)started.
6680.509541966:main thread: Debugging enabled, SIGUSR1 to turn off debugging.
6680.509543921:main thread: initialization completed, transitioning to regular run mode
6680.510497394:main queue:Reg/w0: main queue: entering rate limiter
6680.510557458:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work.
6683.539128966:imuxsock.c: Message from UNIX socket: #3
6683.539552763:imuxsock.c: logmsg: flags 4, pri 5, from 'qa-freebsd', msg Jul  9 11:18:03 root: test
6683.539554719:imuxsock.c: Message has legacy syslog format.
6683.539560865:imuxsock.c: main queue: entry added, size now 1 entries
6683.540008128:imuxsock.c: main queue: EnqueueMsg signaled condition (0)
6683.540010084:imuxsock.c: wtpAdviseMaxWorkers signals busy
6683.540012039:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3
6683.541583748:main queue:Reg/w0: main queue: entering rate limiter
6683.541624256:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries
6683.541658338:main queue:Reg/w0: Called action, logging to builtin-fwd
6683.542046376:main queue:Reg/w0: actionTryResume: iRet: -2007, next retry (if applicable): 1215616710 [now 1215616683]
6683.542110910:main queue:Reg/w0: Called action, logging to builtin-file
6683.542116776:main queue:Reg/w0:  (/var/log/debug.log)
6683.543149310:main queue:Reg/w0: main queue: entering rate limiter
6683.543506897:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work.
^CDoDie called.
6685.813595414:main thread: exiting on signal 2
6685.813628100:main thread: Terminating input threads...
6685.814018931:imuxsock.c: destructor for debug call stack 0x28222000 called
6685.815597065:imklog.c: destructor for debug call stack 0x28234000 called
6685.817005624:main thread: logmsg: flags 5, pri 7, from 'qa-freebsd', msg  [origin software="rsyslogd" swVersion="3.19.10-test1" x-pid="773" x-info="http://www.rsyslog.com"] exiting on signal 2.
6685.817082729:main thread: Message has legacy syslog format.
6685.817090830:main thread: main queue: entry added, size now 1 entries
6685.817535300:main thread: main queue: EnqueueMsg signaled condition (0)
6685.817537256:main thread: wtpAdviseMaxWorkers signals busy
6685.817539491:main thread: Terminating main queue...
6685.817541446:main thread: main queue: initiating worker thread shutdown sequence
6685.817545357:main thread: main queue: trying shutdown of regular workers
6685.817549268:main thread: main queue:Reg: waiting 0ms on worker thread termination, 1 still running
6685.817553459:main thread: main queue:Reg: timeout waiting on worker thread termination
6685.817555414:main thread: main queue: regular shutdown timed out on primary queue (this is OK)
6685.817559326:main thread: main queue: trying immediate shutdown of regular workers
6685.817565472:main thread: main queue:Reg: waiting 1000ms on worker thread termination, 1 still running
6685.819163719:main queue:Reg/w0: main queue: entering rate limiter
6685.819521866:main queue:Reg/w0: main queue:Reg/w0: receiving command 1
6685.819523821:main queue:Reg/w0: main queue:Reg/w0: worker terminating
6685.819588634:main queue:Reg/w0: main queue:Reg: Worker thread 28213310, terminated, num workers now 0
6685.819627186:main queue:Reg/w0: destructor for debug call stack 0x28235000 called
6685.820603288:main thread: waiting for worker main queue:Reg/w0 termination, current state 1
6685.820636532:main thread: main queue:Reg/w0: receiving command 0
6685.820638488:main thread: worker main queue:Reg/w0 has stopped
6685.820640443:main thread: main queue: checking to see if we need to cancel any worker threads of the primary queue
6685.820642399:main thread: main queue:Reg: try canceling worker thread 0
6685.820644355:main thread: main queue: worker threads terminated, remaining queue size 1.
6685.821109777:main thread: main queue: error -7 persisting queue - data lost!
6685.821111732:main thread: Terminating outputs...
6685.821115643:main thread: Freeing log structures.
6685.821565421:main thread: all primary multi-thread sources have been terminated - now doing aux cleanup...
6685.822088113:main thread: file syslogd.c released module 'lmnet', reference count now 2
6685.822090069:main thread:     module lmnet is currently in use by file omfwd.c
6685.822092025:main thread:     module lmnet is currently in use by file conf.c
6685.822095936:main thread: file conf.c released module 'lmnet', reference count now 1
6685.822097891:main thread:     module lmnet is currently in use by file omfwd.c
6685.822100126:main thread: printing users of loadable module lmnet, refcount 1, ptr 0x28214080, type 2
6685.822102082:main thread:     module lmnet is currently in use by file omfwd.c
6685.822104037:main thread: printing users of loadable module builtin-file, refcount 0, ptr 0x282140e0, type 1
6685.823017840:main thread: printing users of loadable module builtin-fwd, refcount 0, ptr 0x28214140, type 1
6685.823019796:main thread: printing users of loadable module builtin-shell, refcount 0, ptr 0x282141a0, type 1
6685.823021752:main thread: printing users of loadable module builtin-discard, refcount 0, ptr 0x28214200, type 1
6685.823023707:main thread: printing users of loadable module builtin-usrmsg, refcount 0, ptr 0x28214260, type 1
6685.823025663:main thread: printing users of loadable module imuxsock.so, refcount 0, ptr 0x282142c0, type 0
6685.823554780:main thread: printing users of loadable module imklog.so, refcount 0, ptr 0x28214320, type 0
6685.823556736:main thread: rsyslog runtime de-initialized, current users 0
6685.823558691:main thread: module lmnet NOT unloaded because it still has a refcount of 1
6685.823560647:main thread: Unloading module builtin-file
6685.823562882:main thread: module lmnet NOT unloaded because it still has a refcount of 1
6685.823564837:main thread: Unloading module builtin-fwd
6685.823566793:main thread: file omfwd.c released module 'lmnet', reference count now 0
6685.823568749:main thread: module 'lmnet' has zero reference count, unloading...
6685.823570704:main thread: Unloading module lmnet
6685.824564685:main thread: Unloading module builtin-shell
6685.824566641:main thread: Unloading module builtin-discard
6685.824568596:main thread: Unloading module builtin-usrmsg
6685.824570552:main thread: Unloading module imuxsock.so
6685.825102742:main thread: Unloading module imklog.so
6685.825529333:main thread: Clean shutdown completed, bye


Is this something very simple I'm just missing between 3.17 and 3.19?

-HKS

oh, udp... I have to admit that I did not test that very well (because it is so lossy). It's probably a bug. There was an interface change and I see that the action is requesting to be suspended. That could be an artifact. Thanks for bringing this up.

mmmhhh... I successfully connected via UDP. Will re-check the debug log...

In the mean time, here's the output when configured to forward via TCP (which worked).

relevant tcpdump lines:

Code: Select all

11:33:38.420548 IP 10.123.0.86.62658 > 10.123.0.97.514: P 147:249(102) ack 1 win 8326 <nop,nop,timestamp 199438 297795934>
E.....@.@.!.
{.V
{.2....h..K:-.... ..@.....
.......^<23>Jul  9 11:33:38 qa-freebsd sudo:     root : TTY=ttyv0 ; PWD=/root ; USER=root ; COMMAND=/bin/test


Once again, I've emailed the -dn output to you since the forum can't handle it. The debugging shows a priority of 23, even though it logs the message with this:

Code: Select all

Jul  9 11:36:41 qa-freebsd sudo:     root : TTY=ttyv0 ; PWD=/root ; USER=root ; COMMAND=/bin/test (local2.notice<149>: 18 5)


-HKS

I think I must see that I get it running on freebsd. Probably it is a platform issue...