Elasticsearch - Add date after index name

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Google Ads


Elasticsearch - Add date after index name

Postby pippo » Fri Jun 16, 2017 11:21 am

Hi,

I installed rsyslog to receive logs from network equipment (switches and routers), and to send it to Elasticsearch. Everything is running smoothly.

The only problem I've got is that there is no date in the index name.

Here is my rsyslog conf :

Code: Select all
module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
#module(load="imklog")   # provides kernel logging support (previously done by rklogd)
#module(load"immark")  # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

# Elasticsearch

module(load="omelasticsearch")

#rsyslog Templates
template(name="ElasticSearchTemplate"
         type="list"
         option.json="on") {
  constant(value="{")
  constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"message\":\"")     property(name="msg")
  constant(value="\",\"host\":\"")        property(name="hostname")
  constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
  constant(value="\",\"syslogtag\":\"")   property(name="syslogtag")
  constant(value="\"}")
}


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/rsyslog-log

*.* action(type="omelasticsearch"
       server="http://localhost:9200"
       serverport="9200"
       searchIndex="rsyslog"
       searchType="rsyslog"
       #dynSearchIndex="on"
       #dynSearchType="on"
       bulkmode="on"
       template="ElasticSearchTemplate"
       errorFile="/var/log/rsyslog-error.log")



I tried to change "searchIndex" and "searchType" to "rsyslog-%{YYYY.MM.dd}" but it gives me an error about the upperscore letters, and if I change to "rsyslog-%{yyyy.mm.dd}" the index name stays like that in Elasticsearch (the "%{yyyy.mm.dd}" isn't replaced with the date).

Is there a way to add the date at the end of the index name?

Thank you in advance.

Pippo
pippo
New
 
Posts: 6
Joined: Fri Jun 16, 2017 11:05 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Elasticsearch - Add date after index name

Postby clayfiske » Wed Sep 06, 2017 6:15 am

Hi Pippo,

You need to use a dynamic index template, like so (in my case the indexes are 'syslog-xxxx' rather than 'rsyslog-xxxx'):

Code: Select all
template(name="elasticidx" type="string" string="syslog-%timestamp:::date-year%.%timestamp:::date-month%.%timestamp:::date-day%")

Then in your action definition, specify it like so:

Code: Select all
           searchIndex="elasticidx"
           dynSearchIndex="on"
           searchType="syslog"


Hope this helps!
clayfiske
New
 
Posts: 3
Joined: Wed Sep 06, 2017 5:29 am

Google Ads



Return to Configuration

Who is online

Users browsing this forum: No registered users and 2 guests

cron