Linux Auditd message parser

Diskussions related to the development of PhpLogCon

Google Ads


Linux Auditd message parser

Postby sire » Wed Nov 18, 2009 11:40 am

Hi All,

I have implemented a custom message parser for Linux auditd logs. Here is the patch. It works :oops: however it's a bit slow comparing to for example webserver logs parser.

I'll try to explain the cause of this slowness. PhpLogCon calls custom message parser functions for every single message. This causes parser class initialization for each message parsed. Auditd log parser uses big (several hundreds of record) lookup tables to convert system call numbers to human-readble names. So auditd parser reads these lookup tables for every message line it parses.

Now I'll describe some details. Lookup table is an array of the following kind:
Code: Select all
$SYSCALL[0] = "restart_syscall";
$SYSCALL[1] = "exit";
$SYSCALL[2] = "fork";
$SYSCALL[3] = "read";
$SYSCALL[4] = "write";
$SYSCALL[5] = "open";
$SYSCALL[6] = "close";
...

I have put this array in a separate php file that is included in auditd message parser php script with the following command:
Code: Select all
private function GetSyscallByNumber($arch, $number) {
   global $gl_root_path;
   include($gl_root_path . "classes/msgparsers/audit.syscalls.$arch.php");
...

I was unable to find a way to statically include lookup tables once for each phpLogCon php script run, not inside custom message parser code. Your help is welcome.

P.S. If this patch fails to apply, try applying my phplogcon-2.7.2-dynamic.patch first. See this thread: http://kb.monitorware.com/dynamic-filenames-feature-bug-t10076.html

Regards,
Sergey Sireskin
Attachments
phpLogCon-auditd-parser.PNG
A screenshot of Auditd parser
phpLogCon-auditd-parser.PNG (78.92 KiB) Viewed 24027 times
phplogcon-2.7.2-auditd.patch.gz
Linux Auditd message parser patch
(14.66 KiB) Downloaded 407 times
sire
Avarage
 
Posts: 18
Joined: Thu Nov 12, 2009 1:19 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Linux Auditd message parser

Postby sire » Tue Nov 24, 2009 10:45 am

Here is an updated Linux Auditd message parser. This version includes some bugfixes and an ability to parse TTY audit data, logged by pam_tty_audit via auditd (see the screenshot below). This patch also extends dynamic files parsing feature as specified in http://kb.monitorware.com/dynamic-filenames-feature-bug-t10076.html.

Regards,
Sergey Sireskin
Attachments
2.PNG
Screenshot
2.PNG (84.31 KiB) Viewed 23924 times
phplogcon-2.7.3-dynamic-auditd-v2.patch.gz
Updated auditd parser
(21.18 KiB) Downloaded 353 times
sire
Avarage
 
Posts: 18
Joined: Thu Nov 12, 2009 1:19 pm

Re: Linux Auditd message parser

Postby sire » Thu Dec 03, 2009 2:49 pm

Hi All,

Here is an update for Linux auditd message parser. It is more compatible with log message formats of different versions of Linux auditd.

Regards,
Sergey Sireskin
Attachments
phplogcon-2.7.3-audit.patch.gz
(18.75 KiB) Downloaded 461 times
sire
Avarage
 
Posts: 18
Joined: Thu Nov 12, 2009 1:19 pm

Re: Linux Auditd message parser

Postby Demos74dx » Wed Jan 16, 2013 1:27 am

THE SYMPTOMS: Whenever you place the word "audit" in the message parsers section of the source edit/build the browser shows a "blank" screen upon reload of Log Analyzer.

After days and days of messing with this I finally got it to work. The culprit is the mappings to associate the account names and passwords with the account id's so root is always account #0 and 0 will show up fine on the parsed message but it wont say "root". I didn't do a thorough fix as I don't have time but its simple enough to look at the log file and find the user account if needed.

THE FIX: go to msgparser.audit.class.php and comment out the entire if-statement that looks like this.

<code>
/*
// If a message was not parsed with ausearch -i - parse it here
if ( !$this->have_ausearch )
{
// UIDs
foreach ( array(SYSLOG_AUDIT_AUID, SYSLOG_AUDIT_UID) as $key ) {
if ( isset($arrArguments[$key]) && ($arrArguments[$key] != '') &&
($arrArguments[$key] >= 0) && ($arrArguments[$key] != 'unset') ) {
if ( $arrArguments[$key] < $this->MAX_UID_RESOLVE ) {
$arr = posix_getpwuid(intval($arrArguments[$key]));
if ( isset($arr['name']) ){
$arrArguments[$key] = $arr['name'];
}
}
}
}
// GIDs
foreach ( array(SYSLOG_AUDIT_GID) as $key ) {
if ( isset($arrArguments[$key]) && ($arrArguments[$key] != '') &&
($arrArguments[$key] >= 0) && ($arrArguments[$key] != 'unset') ) {
if ( $arrArguments[$key] < $this->MAX_UID_RESOLVE ) {
$arr = posix_getgrgid(intval($arrArguments[$key]));
if ( isset($arr['name']) ){
$arrArguments[$key] = $arr['name'];
}
}
}
}
// Arch
if ( isset($arrArguments[SYSLOG_AUDIT_ARCH]) &&
($arrArguments[SYSLOG_AUDIT_ARCH] != '') )
{
$arrArguments[SYSLOG_AUDIT_ARCH] =
$this->GetArchByNumber($arrArguments[SYSLOG_AUDIT_ARCH]);
}
// Sytem call
if ( isset($arrArguments[SYSLOG_AUDIT_SYSCALL]) &&
isset($arrArguments[SYSLOG_AUDIT_ARCH]) &&
($arrArguments[SYSLOG_AUDIT_SYSCALL] != '') &&
($arrArguments[SYSLOG_AUDIT_ARCH] != '') )
{
$arrArguments[SYSLOG_AUDIT_SYSCALL] =
$this->GetSyscallByNumber($arrArguments[SYSLOG_AUDIT_ARCH], $arrArguments[SYSLOG_AUDIT_SYSCALL]);
}
}
*/
</code>
Demos74dx
Avarage
 
Posts: 10
Joined: Fri Jan 11, 2013 12:14 am

Re: Linux Auditd message parser

Postby jenkins » Wed Jan 16, 2013 1:01 pm

is it possible to use a database as source for this parser?

it is working for me but I cant filter the messages...

does any one know hove to get the database format right?
jenkins
New
 
Posts: 2
Joined: Wed Jan 16, 2013 12:31 pm

Re: Linux Auditd message parser

Postby Demos74dx » Mon Jan 21, 2013 6:40 pm

Get on IRC #rsyslog and I might be able to help you out.
Demos74dx
Avarage
 
Posts: 10
Joined: Fri Jan 11, 2013 12:14 am

Re: Linux Auditd message parser

Postby Demos74dx » Mon Jan 21, 2013 6:53 pm

So as far as I know you cannot search via the database. The reason for this is the Parser(as far as I understand it) does the actual parsing on the Log Analyzer side of things, meaning its parsed after it writes to the database directly from the message field. You can have it load into its own source and write to its own database but it is not parsed out in the database. If you can somehow manage to get rsyslog to parse the file when its writing to the database then this would work but you wouldn't have need for the message parser in Log Analyzer.

I have a question for you: Did you need to comment out the lines I specified above to get it to work?
Demos74dx
Avarage
 
Posts: 10
Joined: Fri Jan 11, 2013 12:14 am

Re: Linux Auditd message parser

Postby Demos74dx » Mon Jan 21, 2013 11:00 pm

Okay so after looking into this I've come up with a dirty solution and I'm sure you won't like it. First some background info.

Basically when the Auditd message comes into your centralized rsyslog server you can decide what happens to the file...there are 4 scenarios.

1. You want Auditd to be mixed in with your other logs and have the message parser pick them up and view them individually. I REALLY wouldn't recommend this. Basically here you slam the messages into the same database as everything else using rsyslogs parser and try to get the message parser to pick up the correct messages.

2. You tell rsyslog to parse these messages into a separate database, which log analyzer views as a separate source. The message parser parses these messages quite nicely and they're all separate from your other logs, which I think is nice. The issue here is because the messages are still stored in the database with the incorrect parsing, the actual message is not parsed into its own database and is only being parsed by Log Analyzer in real time. What this means is all search and filter mechanisms that depend on the database layout are lost.

3. You tell rsyslog to parse these messages into a separate LOG FILE, therefore not manipulating the message into a database. You can pick this log file up as a separate source in Log Analyzer and parse them in real time. Since you're not depending on database manipulation to search and filter these functions still work. This is the solution I'm working with right now. Of course you lose any other advantages of having these files stored in a database, I guess is space is not an issue you could tell rsyslog to parse them using both option 2 and 3. Sadly for me space is an issue and I cannot afford to separately store auditd in 2 places just for the advantages of having it in a database. I just have to baby my log files.

4. Someone with more time than me figures out a way to manipulate the rsyslog parser to correctly pick up and parse files into the database in regards to auditd's particular message format. I'm not nearly studied up enough to attempt this, and I just don't have time, otherwise this a the solution I'd love to have.

So it basically comes down to the interaction between Log Analyzer how Log Analyzer reads things and how they're originally organized by rsyslog. The easiest solution is to not use a database for your auditd messages only, just have rsyslog write them to a specific file and have Log Analyzer use that file as a source...you retain all the search and filter functionality.

If someone does come out with a rsyslog parser for auditd I'd love to know about it.
Demos74dx
Avarage
 
Posts: 10
Joined: Fri Jan 11, 2013 12:14 am

Re: Linux Auditd message parser

Postby Demos74dx » Sat Feb 23, 2013 3:48 am

Okay so after even more messing around a little while later, much much more playing with Auditd and LogAnalyzer I'm much more learned on this.

There are two ways you can run the message parser, with ausearch -i functionality, or without. If you are running with ausearch -i functionality then you DO NOT COMMENT OUT THE CODE I MENTION IN THE POST ABOVE.
To turn on ausearch -i functionality you go to /msgparsers/msgparser.audit.class.php and change:
<code>
private $ausearch_path = "/sbin/ausearch";
private $have_ausearch = true;
</code>

Basically the ausearch -i tool changes the unreadable auditd message with UID and GID as numbers into human-readable message with peoples names etc... this tool uses the resources of the current machine. To explain I'll set up a small Scenario. There is one machine called Server-1, this is a central logging server. There is a second machine called Server-2 this is a client server sending logs to Server-1. Jimmy has UID=2 on Server 2 and Teddy has UID=2 on Server 1. Jimmy is a bad guy and decides to delete some files on Server-2, Server-2 sends the auditd logs to Server-1, with something like UID=2(Jimmy) deletes /var/www/html res=success . The message parser(or ausearch -i) then picks up this log and translates this to uid=2(Server-1's /sbin/ausearch settings = Teddy) deleted /html on Server-2! But, JIMMY deleted them on Server-2 and Teddy gets the blame!

So basically if you do not have standardized UID's across ALL YOUR SERVERS you will not get human-readable data that is correct. LDAP is probably the best option here as it standardizes UID's for you.

I would not recommend using ausearch functionality if you do not have LDAP implemented also.

If you don't want to use ausearch functionality then leave as is, comment out the code in my above post, and you'll just have a ton of numbers that wont make too much sense as the message parser wont bother to translate them for you.
Demos74dx
Avarage
 
Posts: 10
Joined: Fri Jan 11, 2013 12:14 am

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 0 guests

cron