elasticsearch-kafka-watch

Basic Overview

This project is a custom watcher for Elasticsearch which works with Apache Kafka by reacting to event in Elasticserch and writing them to Apache Kafka.

Supports 2 types of triggers.

1. Time Based Triggers

This trigger uses crons to trigger when an event should be pushed to Apache Kafka based on the watch configuration.

2. Event Based Triggers

This trigger relies on the IndexListeners and DeleteListeners. Once data is either created or deleted it triggers all watchers that meet criteria and pushes the data to Apache Kafka.

Install

Configuration

Plugin requires a configuration file to know how to connect to Apache Kafka and also how it would use elasticsearch report engine to generate reports. The file elasticsearch-kafka-watch.yml should be place in /path/to/elasticsearch/config folder to be picked by plugin.

Sample Config File

Installation

sudo bin/elasticsearch-plugin install [plugin_name]

Setup And Requirements

Usage

1. Time Based

Create a custom watch with its cron. Events would be generated using the cron. This is written into Apache Kafka. Any worker/consumer listening on Apache Kafka would react to the event.

For sending SMS or Email alerts based on events written in Apache Kafka check out go kafka alert.

Creating a custom watch for a time based cron expects the following parameters:

a.
b.
c.

2. Event Based Triggers

Create a custom watch with and elasticsearch index and query. Once data is written or deleted from the index, it triggers the custom watch to evaluate query to check if there’ll be a hit greater than 0. Once this is positive an event is written to Apache Kafka for consumers/workers listening to react.

For sending SMS or Email alerts based on events written in Apache Kafka check out go kafka alert.

Creating a custom watch for a time based cron expects the following parameters:

a.
b.
c.

3. Report Scheduling

This plugin also works with 2 other plugins to schedule reports using elasticsearch as datasource.

elasticsearch report engine and go kafka alert. The former generates PDF,CSV and HTML reports from elasticsearch using queries. The later sends the reports as email. PDF and CSV reports can be sent as attachments whiles HTML reports can be sent the email body.

Reports can be sent using event based triggers or time based triggers.

Creating a custom watch for a time based cron expects the following parameters:

a.
b.
c.

Supported

Elasticsearch versions supported by this plugin include :

Benchmark Test

Measurement on how fast a trigger is sent to Apache Kafka after indexing and deleting data on Elasticsearch.

System Spec :

Download

Installation

Add the following configuration to your elasticsearch-kafka-watch.yml file located in /path/to/elasticsearch/config

Contribute

Contributions are always welcome! Please read the contribution guidelines first.

Code of Conduct

Please read this.

License

GNU General Public License v3.0