MonitorWare Knowledge Base

Hello,
I just grabbed rsyslog-3.19.7.tar.gz and installed it on
two virtual servers and followed the instructions on rsyslog-
3.19.7.tar.gz. For now, I'm using the certificates from the distrubution. I
believe I've followed the instructions completely, but no love. rsyslog
runs, I can log locally by running logger, but no messages seem to get
logged by the remote server. Doing a tcpdump shows that there is indeed a burst of traffic
on the server whenever I submit a log message from the client.. but
it doesn't appear in /var/log/messages (or anywhere else that I
can see). When I run rsyslog -d on the server and then send a message
from the client, here is what I see (omitting rsyslogd startup
message):

8307.767361000:imtcp.c: New connect on NSD 0x80aec50.
8307.767396000:imtcp.c: GTLS certificate
file: '/root/rsyslog-3.19.7/contrib/gnutls/cert.pem'
8307.767402000:imtcp.c: GTLS key
file: '/root/rsyslog-3.19.7/contrib/gnutls/key.pem'
8307.769674000:imtcp.c: source file nssel.c releasing object
'nsdsel_gtls', ifIsLoaded 1
8307.769697000:imtcp.c: --------<NSDSEL_PTCPcalling select,
active
fds (max 5): 5

That's it. The keys are all there.. it looks like the
message gets to rsyslogd on the server.. but it's rejecting it or something.
I emailed you the output of strace -f rsyslogd -d for one "logger" request
from the client and my config files a couple days ago. (I can't get them to upload here as attachments).
Everything works fine if I remove the TLS stuff from the configs. This is on Ubuntu Gutsy if that's relevant.

Any help would be appreciated Thanks

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

I am having trouble getting TLS to work too, these are the errors I get:
root@mydomain[/etc]# tail -50 /var/log/messages
2008-06-14T23:13:18.385913-04:00 mydomainrsyslogd: [origin software="rsyslogd" swVersion="3.19.5" x-pid="28272" x-info="http://www.rsyslog.com"] restart
2008-06-14T23:13:18.385922-04:00 mydomainrsyslogd:WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c3 as the first rsyslogd option.
2008-06-14T23:13:18.385925-04:00 mydomainrsyslogd:Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad immark
2008-06-14T23:13:18.385927-04:00 mydomainrsyslogd:Warning: backward compatibility layer added to following directive to rsyslog.conf: MarkMessagePeriod 1200
2008-06-14T23:13:18.385930-04:00 mydomainrsyslogd:Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
2008-06-14T23:14:23.015005-04:00 mydomainrsyslogd:error adding our certificate. GnuTLS error -64, message: 'Error while reading file.', key: 'cert.pem', cert: 'key.pem'
2008-06-14T23:14:23.040399-04:00 mydomainrsyslogd:can not read file 'cert.pem': No such file or directory
root@mydomain[/etc]# cat /etc/rsyslog.conf |grep cert
# certificate files - just CA for a client
#$DefaultNetstreamDriverCAFile /usr/share/ssl/certs/ca.pem

Any ideas?

mixtli wrote:Hello,
I just grabbed rsyslog-3.19.7.tar.gz and installed it on
two virtual servers and followed the instructions on rsyslog-
3.19.7.tar.gz. For now, I'm using the certificates from the distrubution. I
believe I've followed the instructions completely, but no ****. rsyslog
runs, I can log locally by running logger, but no messages seem to get
logged by the remote server. Doing a tcpdump shows that there is indeed a burst of traffic
on the server whenever I submit a log message from the client.. but
it doesn't appear in /var/log/messages (or anywhere else that I
can see). When I run rsyslog -d on the server and then send a message
from the client, here is what I see (omitting rsyslogd startup
message):

8307.767361000:imtcp.c: New connect on NSD 0x80aec50.
8307.767396000:imtcp.c: GTLS certificate
file: '/root/rsyslog-3.19.7/contrib/gnutls/cert.pem'
8307.767402000:imtcp.c: GTLS key
file: '/root/rsyslog-3.19.7/contrib/gnutls/key.pem'
8307.769674000:imtcp.c: source file nssel.c releasing object
'nsdsel_gtls', ifIsLoaded 1
8307.769697000:imtcp.c: --------<NSDSEL_PTCPcalling select,
active
fds (max 5): 5

That's it. The keys are all there.. it looks like the
message gets to rsyslogd on the server.. but it's rejecting it or something.
I emailed you the output of strace -f rsyslogd -d for one "logger" request
from the client and my config files a couple days ago. (I can't get them to upload here as attachments).
Everything works fine if I remove the TLS stuff from the configs. This is on Ubuntu Gutsy if that's relevant.

Any help would be appreciated Thanks

I got rid of the SSL TLS errors, by generating my own TLS cert.
Follow the commands on this page:
http://www.rsyslog.com/doc-rsyslog_tls.html

Then I had to run 2 other commands to make the request and cert
certtool --generate-request --load-privkey ../private/ca-key.pem \
--outfile request.pem

and

certtool --generate-certificate --load-request request.pem \
--outfile cert.pem --load-ca-certificate ca.pem \
--load-ca-privkey ../private/ca-key.pem

That starts with no error, and I see traffic going with tcp dump to port 10514 but nothing is getting logged so I am still looking at that. I am going to make a new post in configuration for that sinces its not really an initial installation problem.

I had a look at the configs. As it seems there currently exists a problem with the defaults. However, you should see an error message from rsyslog itself inside the system log (something about wrong certificates and such). Do you have this?

I'll do a lab with the provided config. In the mean time, I suggest that you read along this description about the suggested deployment of recent builds. It is still lacking the config detail, but I'll see that I can get them added ASAP. I'll probably post some config samples here if I can not do that quickly enough. The link is:

http://www.rsyslog.com/doc-rsyslog_secure_tls.html

Thanks,
Rainer

rgerhards wrote:I had a look at the configs. As it seems there currently exists a problem with the defaults. However, you should see an error message from rsyslog itself inside the system log (something about wrong certificates and such). Do you have this?

I'll do a lab with the provided config. In the mean time, I suggest that you read along this description about the suggested deployment of recent builds. It is still lacking the config detail, but I'll see that I can get them added ASAP. I'll probably post some config samples here if I can not do that quickly enough. The link is:

http://www.rsyslog.com/doc-rsyslog_secure_tls.html

Thanks,
Rainer

Thanks! I see traffic going to the remote server from the client on port 10514, but nothing is being logged. Do you have any ideas on how to troubleshoot that?

The best thing is always to run BOTH the client and the server rsyslog in debug mode (-dn). This is quite verbose and should tell you what is going on. If nothing is received in these logs, it is definitely outside of rsyslog, e.g. a firewall problem. But I think the problem is related to authentication defaults. I just wonder why nothing is logged (but my lab is ready soon, my current work environment was somewhat unusable as a lab right now).

I think that adding an

$ActionSendStreamDriverAuthMode anon

on the client side and

$InputTCPServerStreamDriverAuthMode anon

on the server side will probably solve the issue. But if it does, keep in mind that the setup is insecure and have a look at the link I posted.

Rainer

OK, it looks like I can reproduced the problem. I'll keep you posted.

Rainer

I see there absence of the client's private key is not correctly handled. However, in my text log, I receive this message:

Code: Select all

2008-06-16T15:07:25.546608+02:00 rgf9dev rsyslogd:error adding our certificate. GnuTLS error -64, message: 'Error while reading file.', key: 'cert.pem', cert: 'key.pem'

Don't you see a similar message? This is what puzzles me most currently...

Rainer

OK, so here is the work-around for the (insecure) anonymous case (where there is no authentication of the peers):

Server:

Code: Select all

$ModLoad /home/rger/proj/rsyslog/plugins/imuxsock/.libs/imuxsock.so
$ModLoad /home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp
$ModLoad /home/rger/proj/rsyslog/plugins/imudp/.libs/imudp.so

*.* -/home/rger/proj/rsyslog/logfile

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/contrib/rger/ca.pem
$DefaultNetstreamDriverCertFile /home/rger/proj/rsyslog/contrib/rger/cert.pem
$DefaultNetstreamDriverKeyFile /home/rger/proj/rsyslog/contrib/rger/key.pem
#$DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem
#$DefaultNetstreamDriverCertFile /path/to/contrib/gnutls/cert.pem
#$DefaultNetstreamDriverKeyFile /path/to/contrib/gnutls/key.pem

$ModLoad /home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp # load listener

$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerRun 10514 # start up listener at port 10514


Client:

Code: Select all

$ModDir /home/rger/proj/rsyslog/.libs/
$ModLoad /home/rger/proj/rsyslog/plugins/imuxsock/.libs/imuxsock.so

# certificate files - just CA for a client
#$DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem
$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/contrib/rger/ca.pem

# work-around for rsyslog 3.19.7 bug
# instead of the work-around, it is prefferred to set up a secure system
# where each peer is authenticated
$DefaultNetstreamDriverCertFile /home/rger/proj/rsyslog/contrib/rger/cert.pem
$DefaultNetstreamDriverKeyFile /home/rger/proj/rsyslog/contrib/rger/key.pem
# end work-around

# set up the action
$DefaultNetstreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS for the connection
$ActionSendStreamDriverAuthMode anon
*.* @@172.19.2.7:10514 # send (all) messages
*.* /home/rger/proj/rsyslog/logfile


Note that both configs are from my lab environment. The cert pathes do not match yours (or those from the doc set). Note that currently the client always must have a certificate and matching private key, because of a bug that disables truly anonymous operations. I'll fix that bug and update the doc set. Also, it is strongly advised to use a more secure configuration, for which I will also update the doc set. So in short: please keep following this thread.

Feedback is appreciated.

Rainer

Yes, I am seeing that error too when I do the verbose debugging:
4162.665985035:main queue:Reg/w0: unexpected GnuTLS error -64 in nsd_gtls.c:481: Error while reading file.
4162.665998310:main queue:Reg/w0: Called LogError, msg: error adding our certificate. GnuTLS error -64, message: 'Error while reading file.',
key: 'cert.pem', cert: 'key.pem'

I will try your work around now.

I generated the cert on the client, but I am having difficutly getting that to work, I even turn the firewall off on the client and server.

This is the ca.pem, that you copied from the server right?
$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/contrib/rger/ca.pem

Also, in the log on the server I'm getting this error:
2008-06-16T21:51:33.000594-05:00 lds143 rsyslogd:Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
2008-06-16T21:51:33.000653-05:00 lds143 rsyslogd:could not load module '/usr/local/lib/rsyslog/imuxsock.so', rsyslog error -2015

Even though imuxsock is there:

root@server1 [/var/log]# ll /usr/local/lib/rsyslog/imuxsock.so
-rwxr-xr-x 1 root root 28416 Jun 3 23:50 /usr/local/lib/rsyslog/imuxsock.so*

Is it suppose to be 755?

abefroman wrote:Yes, I am seeing that error too when I do the verbose debugging:
4162.665985035:main queue:Reg/w0: unexpected GnuTLS error -64 in nsd_gtls.c:481: Error while reading file.
4162.665998310:main queue:Reg/w0: Called LogError, msg: error adding our certificate. GnuTLS error -64, message: 'Error while reading file.',
key: 'cert.pem', cert: 'key.pem'

I will try your work around now.

You should also see this message inside your logs. I think it is logged with syslog.err, maybe you have not logged these messages. I suggest you do, because otherwise you don't see what happens when something goes wrong.

abefroman wrote:I generated the cert on the client, but I am having difficutly getting that to work, I even turn the firewall off on the client and server.

This is the ca.pem, that you copied from the server right?
$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/contrib/rger/ca.pem

Yes, but this is not for the server, but the CA cert itself. It needs to be present on each system. The CA private key MUST NOT be present on ANY system, you just need it when you generate new machine certificates.

abefroman wrote:Also, in the log on the server I'm getting this error:
2008-06-16T21:51:33.000594-05:00 lds143 rsyslogd:Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
2008-06-16T21:51:33.000653-05:00 lds143 rsyslogd:could not load module '/usr/local/lib/rsyslog/imuxsock.so', rsyslog error -2015

This is because you are using rsyslog in backward compatibility mode and have legacy options in the command line (-r at least). Please read this for an explanation:

http://www.rsyslog.com/doc-v3compatibility.html

The bottom line is that you should move over to native v3 mode. In any case, what you see is not an error, it is just an indication that a modules is being loaded twice (a nicer error message is on the todo list ).

abefroman wrote:Even though imuxsock is there:

root@server1 [/var/log]# ll /usr/local/lib/rsyslog/imuxsock.so
-rwxr-xr-x 1 root root 28416 Jun 3 23:50 /usr/local/lib/rsyslog/imuxsock.so*

Is it suppose to be 755?

Actually, that's much depending on the packager. From the rsyslog perspective it is just important that the module is executable for the user under which rsyslog runs (usually root). So 500 should be fine.

If you do not receive messages, post (or mail me if it is sensitive) a server AND client debug log. To obtain that, run rsyslog interactive with -dn options added. Advanced options are described here:

http://www.rsyslog.com/doc-debug.html

But these are usually not needed.

Rainer

rgerhards wrote:

abefroman wrote:This is because you are using rsyslog in backward compatibility mode and have legacy options in the command line (-r at least). Please read this for an explanation:

http://www.rsyslog.com/doc-v3compatibility.html

The bottom line is that you should move over to native v3 mode. In any case, what you see is not an error, it is just an indication that a modules is being loaded twice (a nicer error message is on the todo list ).

Rainer

Odd, Im using the standard rsyslogd command to start it and rsyslogd -dn to debug. Does it think I have legacy options because of the way I have the modules in the config file?

root@a [~]# rsyslogd -v
rsyslogd 3.19.5, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: Yes
FEATURE_NETZIP (message compression): Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
Runtime Instrumentation (slow code): No

See http://www.rsyslog.com for more information.
root@a[~]#

Question, on your server config file, you have:
$ModLoad /home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp
that should be
$ModLoad /home/rger/proj/rsyslog/plugins/imtcp/.libs/imtcp.so
right?