Logstash: How to trigger an alarm or event

The ELK stack (ElasticSearch, LogStash and Kibana) is a great tool to centralize log monitoring. The following configuration is what I use for triggering an alarm or event when something that needs immediate attention happens on the log file.

The trigger/alarm is quite easy to implement and is done by changing the configuration file for the logstash instance that gets data from the Redis database and outputs the data into ElasticSearch.

For example, the base configuration file is something like this:

input {
  redis {
    host => "127.0.0.1"
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"
    # We use the 'json' codec here because we expect to read
    # json events from redis.
    codec => json
  }
}

output {
   #stdout { debug => true debug_format => "json"}
   elasticsearch {
   host => "127.0.0.1"
   }
}

This example comes out straight from the Logstash documentation.

So, what if I want to do something when some event/string shows up on the log files? We know that everything passes through this phase (REDIS -> ElasticSearch)

The answer is that all we needed to do is to change the output section to the following:

output {
  #stdout { debug => true debug_format => "json"}
  elasticsearch { 
    host => "127.0.0.1"
  }

  if "com.megacorp.security.exception.Exception" in [message] 
    { 
      exec { 
        command => "/home/megacorp/monit/bin/notify.sh Error_message &" 
      } 
    }
}

So all is needed is to add the if clause, and the condition behaves as grep into the JSON message field.
So we can add this section several times to call any external command with exec, or send mail, or anything that logstash supports.  Just make sure that if using the exec plugin, the command has the & operator at the end to run on the background and not lock the logstash agent.
Also the [message] JSON component can be changed to anything that the JSON provides, like [tags] or [type].

 

Advertisements

2 thoughts on “Logstash: How to trigger an alarm or event

  1. Hello, iam very new to the ELK stack. I followed a guide on digitalocean on how to setup elk. It works fine and all but just as you I need to get some alarms. Could you maybe try to elaborate abit. I dont understand what config file i need to put all code in.

    Sorry for bad english and bad knowlede. Iam kinda new to linux and elk.

    Regards Jonatan

  2. Hi: Logstash runs in three instances: As an agent, as the log shipper, and to provide the Kibana web interface.

    For that we need to provide a configuration file for the first two functions, the agent and the log shipper. This last one picks up data that is sent from the agents into the Redis database, and ships it to the Elastic Search Engine.

    So the log shipper uses the above configuration file that I named indexer.conf. The agents use a file that I named agent.conf, but in this case since all logs passes through the indexer, I’ve added the notification here.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s