Logstash Tutorial : A complete guide for the beginners how to index data from logstash to Elasticsearch and Kibana

Logstash Tutorial : A complete guide for the beginners how to index data from logstash to Elasticsearch and Kibana

   December 26, 2019       Codez Tech

---

Logstash is an open-source, server-side data processing pipeline that ingests data from many sources at once, transforms it, and then sends it to your favourite "stash" — most commonly Elasticsearch.

Logstash

The below topics are covered in this blog -

1) Overview of Logstash
2) What is Logstash?
3) Installing Logstash
4) Downloading a sample dataset
5) How to run Logstash
6) A complete simple.conf example

1. Overview of Logstash

Developed by: Elastic NV. Logstash is one of the three core components of the Elastic (ELK) stack, alongside Elasticsearch and Kibana.

2. What is Logstash?

Logstash is a lightweight, open-source, server-side data processing pipeline that lets you collect data from a variety of sources, transform it on the fly, and send it to your chosen destination. It is most often used as the data pipeline for Elasticsearch. Thanks to its tight Elasticsearch integration, powerful log-processing capabilities, and a large ecosystem of pre-built plugins, Logstash is a popular choice for loading data into Elasticsearch.

The Logstash Pipeline — Inputs, Filters, Outputs

3. Installing Logstash

Download Logstash from the official Elastic site. Always match the Logstash version to your Elasticsearch version:

https://www.elastic.co/downloads/logstash

4. Download a Dataset to Import into Elasticsearch

You can grab a free sample dataset from Kaggle. This tutorial uses the IBM HR Analytics employee attrition dataset:

https://www.kaggle.com/datasets

5. How to Run Logstash

A Logstash pipeline has three stages: input (where the data comes from), filter (how it is parsed and transformed), and output (where it goes). At its simplest:

input {
  stdin { }
}

filter { }

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Save the file with a .conf extension, then run it from your Logstash folder:

bin/logstash -f simple.conf     # simple.conf is your config file name

Official configuration reference: Logstash configuration guide.

6. A Complete simple.conf Example

This config reads a CSV, converts the numeric columns to integers, and indexes the result into Elasticsearch. Update the path to point at your own CSV file.

input {
  file {
    path => "/path/to/your/employee.csv"
    start_position => "beginning"
    sincedb_path => "NUL"
  }
}

filter {
  csv {
    separator => ","
    columns => [ "Age","Attrition","BusinessTravel","DailyRate","Department",
      "DistanceFromHome","Education","EducationField","EmployeeCount",
      "EmployeeNumber","EnvironmentSatisfaction","Gender","HourlyRate",
      "JobInvolvement","JobLevel","JobRole","JobSatisfaction","MaritalStatus",
      "MonthlyIncome","MonthlyRate","NumCompaniesWorked","Over18",
      "OverTime","PercentSalaryHike","PerformanceRating",
      "RelationshipSatisfaction","StandardHours","StockOptionLevel",
      "TotalWorkingYears","TrainingTimesLastYear","WorkLifeBalance",
      "YearsAtCompany","YearsInCurrentRole","YearsSinceLastPromotion",
      "YearsWithCurrManager" ]
  }

  mutate {
    convert => {
      "Age" => "integer"
      "DailyRate" => "integer"
      "DistanceFromHome" => "integer"
      "Education" => "integer"
      "EmployeeCount" => "integer"
      "EmployeeNumber" => "integer"
      "EnvironmentSatisfaction" => "integer"
      "HourlyRate" => "integer"
      "JobInvolvement" => "integer"
      "JobLevel" => "integer"
      "JobSatisfaction" => "integer"
      "MonthlyIncome" => "integer"
      "MonthlyRate" => "integer"
      "NumCompaniesWorked" => "integer"
      "PercentSalaryHike" => "integer"
      "PerformanceRating" => "integer"
      "RelationshipSatisfaction" => "integer"
      "StandardHours" => "integer"
      "StockOptionLevel" => "integer"
      "TotalWorkingYears" => "integer"
      "TrainingTimesLastYear" => "integer"
      "WorkLifeBalance" => "integer"
      "YearsAtCompany" => "integer"
      "YearsInCurrentRole" => "integer"
      "YearsSinceLastPromotion" => "integer"
      "YearsWithCurrManager" => "integer"
    }
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "employee"
  }
  stdout { }
}

Clone the full project:

git clone https://bitbucket.org/atique1224/youtube_logstash_tutorial.git

7. Video Walkthrough

---

About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub