MonitorWare Knowledge Base

Hello, I'm trying to setup a central rsyslog on a Ubuntu 2.6.22 on a sparc64 system to collect eventlog from various hosts, many of them W2k3 server, and a few different Linux flavours (Ubuntu, RedHat...).
Everything is working fine, except this:
the eventlog msg from Windows has a wrong (?) format, so it's misunderstood by rsyslog, even if it is displayed by phpLogCon.
I've tried Snare for Windows from one w2k3 and Eventlog to Syslog Service (Purdue), both fails.
I've attached two screenshot (from the same event): evtLog593_Win.jpg (from Windows point of view) and evtLog593_Lnx.jpg (from rsyslog/phpLogCon pov), to help understand what my problem is.
Where can I find a way to tell rsyslog the exact format? Or to W2k3 to change it's output format?
TIA
Luigi VISINTIN

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

The eventlog is not syslog-compliant, so it doesn't use the same format. Purdue's Syslog to Eventlog has worked fairly well for me, but it's not strictly compliant. You can work with the rsyslog templates to try forcing the messages into a format understood by phplogcon (not familiar with that software, so I can't speak to what it needs). We might be able to help if you can copy+paste a flawed message as it is received by rsyslog (ie, from your logfile rather than from phplogcon).

Alternatively, you can install one of MonitorWare's Windows Event reporting software packages. I've not had a chance to test any of them yet, but since they're made by the same company as rsyslog and phplogcon, they'll probably do what you need.

-HKS

If it helps, this is what I'm using as a default logging template for remote logs:

$template remote_default, "%timegenerated% %fromhost% %syslogtag%%msg:::drop-last-lf%\n"


-HKS

Hi hkspvt, thx for your quick answer. I'm a newbie on rsyslog, but I'll try to apply such template...
Cheers
Luigi

Hi hkspvt,
following your suggestion I made a few test, but still something does'nt work...
1 I've added to "/etc/syslog.conf" the following (one at a time):
$template remote_default, "%timegenerated% %fromhost% %syslogtag%%msg:::drop-last-lf%\n"
$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,%syslogtag%%msg%\n"

2 when only the first of those to templates was set phpLogCon made a small enhancement: the record, coming from a W2k3 Stdalone server, was shown on the list, but only a few fields were correctly populated. Here is the '/etc/var/syslog' record (those ??? should be, I think, the host name, maybe I should use %HOSTNAME% instead of %fromhost%?):

2008-08-21T13:25:14.287461+02:00 ??? Security: 592: NT AUTHORITY\SYSTEM: Èstato creato un nuovo processo: Nuovo ID del processo: 5956 Nome del file immagine: C:\WINDOWS\system32\verclsid.exe ID del processo creatore: 5188 Nome utente: NBVISINTIN$ Dominio: TWINHAWKS ID di accesso: (0x0,0x3E7)

3 when only the second template row was set the results was (reading from "/etc/var/messages"):
2008-08-21T13:06:24+02:00 bl6.istge.priv MSWinEventLog#0111#011Security#011888515#011Thu Aug 21 13:06:24 2008#011861#011Security#011SYSTEM#011User#011Failure Audit#011BL6#011Detailed Tracking#011#011The Windows Firewall has detected an application listening for incoming traffic. Name: - Path: C:\Program Files\Snare\SnareCore.exe Process identifier: 3076 User account: SYSTEM User domain: NT AUTHORITY Service: Yes RPC server: No IP version: IPv4 IP protocol: TCP Port number: 6161 Allowed: No User notified: No #011887389

but still the fileds were not all populated in the phpLogCon detail view.

Can you explain me:
a) why those records are going to different files (/etc/var/messages or /etc/var/syslog)?
b) I'm correctly understanding that "#011" is a notation 'equivalent' to ASCII(11), meaning this is the delimiter of each field in the record?
Thanks a lot again for what help you can provide.
Luigi

Sorry, in my previous post I forgot to attach two snapshots of phpLogCon detail view.
Here they are.
The data is not really the same, but the appearance is the same, so I think you can understand what field was missing...
Cheers
Luigi

Can you paste in your entire /etc/syslog.conf file?

-HKS

Of course... here it is, and again thx...
Luigi
---------------------------------------
# /etc/syslog.conf Configuration file for syslogd.
#
# For more information see syslog.conf(5)
# manpage.
$ModLoad immark.so # provides --MARK-- message capability
#$ModLoad imudp.so # provides UDP syslog reception
$ModLoad imtcp.so # provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
$ModLoad ommysql.so # provides mysql writing support

# UDP Syslog Server:
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514
#
# Copiati dal sito rsyslog.com/doc-ommail.html
#$ModLoad ommail
$ActionMailSMTPServer mailcluster.istge.it
$ActionMailFrom rsyslog@istge.it
$ActionMailTo giovanni.depascalis@istge.it; visintin@inrete.it
$template mailSubject,"test del subject sulla macchina %hostname%"
$template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
$ActionMailSubject mailSubject
# make sure we receive a mail only once in six
# hours (21,600 seconds )
$ActionExecOnlyOnceEveryInterval 600
# the if ... then ... mailBody mus be on one line!
if $msg contains 'service entered the stopped state' then mmail:;mailBody
#
#-- aggiunti per formato log da W2k3 (LV 2008-08-21)
# $template remote_default, "%timegenerated% %fromhost% %syslogtag%%msg:::drop-last-lf%\n"
# $template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,%syslogtag%%msg%\n"
#--
#
#
# First some standard logfiles. Log by facility.
#

auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err

# Logging for INN news system
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice

#
# Some `catch-all' logfiles.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg *

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole

--------------------------------------------------------------
(end of file)

a) The first log probably had a severity of WARNING or NOTICE, and was dumped into both messages and syslog. The second probably had a severity of ERROR, so it was only caught by syslog. I prefer to use the messages file as a sort of one-glance "What was happening of significance around this time?" log, so here's my selector:

*.notice;authpriv.info;kern.debug;lpr.info;mail.none

Whatever you choose to do with it, be aware of what will and will not be logged to that file.


b) I believe you're correct, but I'm not sure why it's showing up in your logs. It appears that your commas are being interpreted as #011 - perhaps because the parser sees them as %,%? This may be a bug for Rainer to take a look at.

The ??? is probably showing up because reverse DNS is not configured for the source IP.

The following template will not actually correct the display problems in PHPLogCon and your syslogs, but it should help clarify what fields show what values. Can you run this, then post a log message or two?
$template test, "TIME GEN: %timegenerated:::date-rfc3339% HOSTNAME: %hostname% FROMHOST: %fromhost% FROMIP: %fromhost-ip% FACILITY: %syslogfacility% PRIORITY: %syslogpriority% TAG: %syslogtag% MSG: %msg%\n"

-HKS

Hi HKS,
I'm glad you have the time to help me...
My only problem now is that next two workdays I'll be on a customer in a different town, so I can't do those test before midweek, but I'll do it and come back with results.
In the meantime, just for info:
1) I'm using SNARE for Windows as a 'sending' agent, and I've specified in SNARE cfg that ALL type of events should go to rsyslog, so I can filter on a central point instead than on any single server, and this is still the study phase, before going prod. This is case 3/b on my previous post.
2) The difference in data could also depend from the fact that both sender were W2k3, but one is NOT JOINED to the Win domain, and maybe this is the reason of missing hostname, no reverse DNS, it's a stdalone server in a workgroup, and this has "Purdue's Syslog to Eventlog" as sending agent (case 2/a).
Ok, I'll try to test as you suggest and come back ASAP.
Thx for now, and have a nice weekend
Luigi

Hello, I'm here again...
I did some test, but results are not so clear...
At bottom you'll find:
1 the current rsyslog.conf (I hope I did understand correctly your suggestions... if not pls tell me)
2 a text file with 3 records from 'test.log', those are as follow:
the first record comes from the 'SNARE' agent, his name is bl6.istge.it (correct), but I don't see the IP...
the second is a local msg (from the same ubuntu server where rsyslog is running)
the third comes from the 'not joined' server, (with Purdue EventLog agent) where the only info on the sending server is "???" (neither name nor ip)
Another thing I don't understand is: why only 'security' evt log record are coming from Purdue? Is it any change I shall do on that machine to make it send ALL event logs? I didin't find any config option... maybe I shall install Snare also on this?
Thx again for your precious help
Luigi
------------------------------ rsyslog.conf ----------------------------
# /etc/syslog.conf Configuration file for syslogd.
#
# For more information see syslog.conf(5)
# manpage.
$ModLoad immark.so # provides --MARK-- message capability
#$ModLoad imudp.so # provides UDP syslog reception
$ModLoad imtcp.so # provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
$ModLoad ommysql.so # provides mysql writing support

# UDP Syslog Server:
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514
#
# Copiati dal sito rsyslog.com/doc-ommail.html
#$ModLoad ommail
$ActionMailSMTPServer mailcluster.istge.it
$ActionMailFrom rsyslog@istge.it
$ActionMailTo giovanni.depascalis@istge.it; visintin@inrete.it
$template mailSubject,"test del subject sulla macchina %hostname%"
$template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
$ActionMailSubject mailSubject
# make sure we receive a mail only once in six
# hours (21,600 seconds )
$ActionExecOnlyOnceEveryInterval 600
# the if ... then ... mailBody mus be on one line!
if $msg contains 'service entered the stopped state' then mmail:;mailBody
#
# formato log da Windows 2003
#$template remote_default, "%timegenerated% %fromhost% %syslogtag%%msg:::drop-last-lf%\n"
#$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,%syslogtag%%msg%\n"
#
$template test, "TIME GEN: %timegenerated:::date-rfc3339% HOSTNAME: %hostname% FROMHOST: %fromhost% FROMIP: %fromhost-ip% FACILITY: %syslogfacility% PRIORITY: %syslogpriority% TAG: %syslogtag% MSG: %msg%\n"

#----------------------------------------------------
# da mail forum
#*.notice;authpriv.info;kern.debug;lpr.info;mail.none
#
*.* -/var/log/test.log

# |
# | #
# | #
# | # First some standard logfiles. Log by facility.
# | #
# |
# | auth,authpriv.* /var/log/auth.log
# | *.*;auth,authpriv.none -/var/log/syslog
# | #cron.* /var/log/cron.log
# | daemon.* -/var/log/daemon.log
# | kern.* -/var/log/kern.log
# | lpr.* -/var/log/lpr.log
# | mail.* -/var/log/mail.log
# | user.* -/var/log/user.log
# |
# | #
# | # Logging for the mail system. Split it up so that
# | # it is easy to write scripts to parse these files.
# | #
# | mail.info -/var/log/mail.info
# | mail.warn -/var/log/mail.warn
# | mail.err /var/log/mail.err
# |
# | # Logging for INN news system
# | #
# | news.crit /var/log/news/news.crit
# | news.err /var/log/news/news.err
# | news.notice -/var/log/news/news.notice
# |
# | #
# | # Some `catch-all' logfiles.
# | #
# | *.=debug;\
# | auth,authpriv.none;\
# | news.none;mail.none -/var/log/debug
# | *.=info;*.=notice;*.=warn;\
# | auth,authpriv.none;\
# | cron,daemon.none;\
# | mail,news.none -/var/log/messages
# |
# | #
# | # Emergencies are sent to everybody logged in.
# | #
# | *.emerg *
# |
# | #
# | # I like to have messages displayed on the console, but only on a virtual
# | # console I usually leave idle.
# | #
# | #daemon,mail.*;\
# | # news.=crit;news.=err;news.=notice;\
# | # *.=debug;*.=info;\
# | # *.=notice;*.=warn /dev/tty8
# |
# | # The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# | # you must invoke `xconsole' with the `-file' option:
# | #
# | # $ xconsole -file /dev/xconsole [...]
# | #
# | # NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# | # busy site..
# | #
# | daemon.*;mail.*;\
# | news.err;\
# | *.=debug;*.=info;\
# | *.=notice;*.=warn |/dev/xconsole
# |

--------------------------------------------------------------------------
------------------------------ test.log ---------------------------------
2008-08-29T12:06:06+02:00 bl6.istge.priv MSWinEventLog#0111#011Security#011912097#011Fri Aug 29 12:06:06 2008#011540#011Security#011MONDOEDP$#011User#011Success Audit#011BL6#011Logon/Logoff#011#011Successful Network Logon: User Name: MONDOEDP$ Domain: ISTGE Logon ID: (0x6,0x93AF88FB) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {7cfc5f52-ce3d-328b-5f68-349b03fbaca9} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.1.0.153 Source Port: 0 #011910865
2008-08-29T12:17:01.364375+02:00 ubuntu CRON[26101]: pam_unix(cron:account): account root has password changed in future
2008-08-29T12:27:05.541984+02:00 ??? Security: 592: NBVISINTIN\Administrator: È stato creato un nuovo processo: Nuovo ID del processo: 4468 Nome del file immagine: C:\WINDOWS\system32\cmd.exe ID del processo creatore: 3716 Nome utente: Administrator Dominio: NBVISINTIN ID di accesso: (0x0,0x30B97)
----------------------------- END ---------------------------------------

Welcome back.

The log entries your provided have not had the template applied to them. To apply a template to a log ouptut, the action needs to be followed by a semicolon (;) and then the template name. So, in this case, the line:
*.* -/var/log/test.log

Needs to read:
*.* -/var/log/test.log;test

You can also set the default template (after you've defined the template in question) with a line like:
$ActionFileDefaultTemplate test

This may be why the other templates i suggested did not seem to make a difference - but apply this one, and let's make sure the information is all presented correctly.

-HKS

Thx, I didn't very well my homework... sorry.
I'll try this and come back...
Cheers
Luigi

Yup... that's a miracle, as we say here in Italy... if you tell the right thing it works...
Ok, thx, now I'll do a few test to see various type of events from different servers... I'm also trying to understand (from rfc3164...) how the different Windows Events are managed...
I'll be back again with questions, I'm sure, but you were really helpful.
Thanks a lot for now
Luigi

Glad to hear it's working.

I wouldn't count on the RFCs for a whole lot of information on Windows Event Logs. MS completely disregards the syslog standard in favor of their proprietary Event Log system, hence the need for translation apps like Eventlog to Syslog. Purdue gives a pretty simple explanation of how their app does the translation with respsect to facility and severity/priority:

The Eventlog to Syslog service forwards messages to the syslog server with a facility of DAEMON and priority of ERR, WARNING, or NOTICE. It is possible to use a different syslog facility than DAEMON by specifying a -f switch during the installation.

FWIW, I'm getting every Windows event sent over to my main syslog server using that program, regardless of the Event Log section it's in.

-HKS