Logstash Tutorial : A complete guide for the beginners how to index data from logstash to Elasticsearch and Kibana
Logstash is an open-source, server-side data processing pipeline that ingests data from many sources at once, transforms it, and then sends it to your favourite "stash" — most commonly Elasticsearch.
 |
| Logstash |
The below topics are covered in this blog -
1) Overview of Logstash
2) What is Logstash?
3) Installing Logstash
4) Downloading a sample dataset
5) How to run Logstash
6) A complete simple.conf example
1. Overview of Logstash
Developed by: Elastic NV. Logstash is one of the three core components of the Elastic (ELK) stack, alongside Elasticsearch and Kibana.
2. What is Logstash?
Logstash is a lightweight, open-source, server-side data processing pipeline that lets you collect data from a variety of sources, transform it on the fly, and send it to your chosen destination. It is most often used as the data pipeline for Elasticsearch. Thanks to its tight Elasticsearch integration, powerful log-processing capabilities, and a large ecosystem of pre-built plugins, Logstash is a popular choice for loading data into Elasticsearch.
 |
| The Logstash Pipeline — Inputs, Filters, Outputs |
3. Installing Logstash
Download Logstash from the official Elastic site. Always match the Logstash version to your Elasticsearch version:
https://www.elastic.co/downloads/logstash
4. Download a Dataset to Import into Elasticsearch
You can grab a free sample dataset from Kaggle. This tutorial uses the IBM HR Analytics employee attrition dataset:
https://www.kaggle.com/datasets
5. How to Run Logstash
A Logstash pipeline has three stages: input (where the data comes from), filter (how it is parsed and transformed), and output (where it goes). At its simplest:
input {
stdin { }
}
filter { }
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}Save the file with a .conf extension, then run it from your Logstash folder:
bin/logstash -f simple.conf # simple.conf is your config file nameOfficial configuration reference: Logstash configuration guide.
6. A Complete simple.conf Example
This config reads a CSV, converts the numeric columns to integers, and indexes the result into Elasticsearch. Update the path to point at your own CSV file.
input {
file {
path => "/path/to/your/employee.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => [ "Age","Attrition","BusinessTravel","DailyRate","Department",
"DistanceFromHome","Education","EducationField","EmployeeCount",
"EmployeeNumber","EnvironmentSatisfaction","Gender","HourlyRate",
"JobInvolvement","JobLevel","JobRole","JobSatisfaction","MaritalStatus",
"MonthlyIncome","MonthlyRate","NumCompaniesWorked","Over18",
"OverTime","PercentSalaryHike","PerformanceRating",
"RelationshipSatisfaction","StandardHours","StockOptionLevel",
"TotalWorkingYears","TrainingTimesLastYear","WorkLifeBalance",
"YearsAtCompany","YearsInCurrentRole","YearsSinceLastPromotion",
"YearsWithCurrManager" ]
}
mutate {
convert => {
"Age" => "integer"
"DailyRate" => "integer"
"DistanceFromHome" => "integer"
"Education" => "integer"
"EmployeeCount" => "integer"
"EmployeeNumber" => "integer"
"EnvironmentSatisfaction" => "integer"
"HourlyRate" => "integer"
"JobInvolvement" => "integer"
"JobLevel" => "integer"
"JobSatisfaction" => "integer"
"MonthlyIncome" => "integer"
"MonthlyRate" => "integer"
"NumCompaniesWorked" => "integer"
"PercentSalaryHike" => "integer"
"PerformanceRating" => "integer"
"RelationshipSatisfaction" => "integer"
"StandardHours" => "integer"
"StockOptionLevel" => "integer"
"TotalWorkingYears" => "integer"
"TrainingTimesLastYear" => "integer"
"WorkLifeBalance" => "integer"
"YearsAtCompany" => "integer"
"YearsInCurrentRole" => "integer"
"YearsSinceLastPromotion" => "integer"
"YearsWithCurrManager" => "integer"
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "employee"
}
stdout { }
}Clone the full project:
git clone https://bitbucket.org/atique1224/youtube_logstash_tutorial.git
