Saturday, September 12, 2020

How to Increase the twitter followers instantly free


This tutorial is all about making a Twitter bot using JavaScript / NodeJS, which helps you retweet certain hashtags along with a like. It helps you gain followers on your Twitter account. It will not happen all of a sudden — you will start seeing results after about 2 weeks.

How to gain Twitter followers using a Twitter bot
How to gain Twitter followers

The below topics are covered in this blog -

1) How to increase Twitter followers?
2) How to gain Twitter followers?
3) How to get Twitter free unlimited retweet?
4) Free Twitter followers
5) How to gain followers on Twitter organically?
6) How to increase real Twitter followers?
7) Serverless hosting
8) Serverless deployment
9) Making a Twitter bot with JavaScript
10) Making a Twitter bot with NodeJS
11) AWS Lambda deployment

1. Apply for a Twitter Developer Account

First, you have to apply for a Twitter developer account if you haven't already. It takes 2–3 days to get activated. Once it is active:

Twitter developer account dashboard with consumer keys and access tokens
Twitter Developer Account

2. Add Your Keys to the Config

You will get your consumer keys and authentication tokens. Grab your keys and tokens and paste them into the config file at config/default.json.

Twitter bot built with JavaScript and NodeJS source code
Twitter Bot with JavaScript

3. Run the Bot

Install the dependencies and run the handler:

npm install
node handler.js

For code-level access, I am providing the GitHub link below — you can clone it from there. If you like my article, please follow me on GitHub. If you have any doubts, watch my YouTube video.

GitHub - https://github.com/codeztech-atique/twitter-bot

4. Deploy to AWS Lambda

Once the code is ready, I deploy this JavaScript function as a Lambda on AWS and trigger it every 2 hours. Why Lambda? Because AWS Lambda is serverless — there is no downtime, and I don't pay any extra amount to AWS. I only pay for what I consume. For me it works out very cheap, around 0.15 dollars a month or less. More details in the YouTube video below.

5. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Monday, January 27, 2020

Integrating Jenkins with Slack and Email Notification


Continuous Integration is a process in which all development work is integrated as early as possible, with the resulting artefacts automatically built and tested. This blog walks through integrating Jenkins with Slack and email notifications, so your team is notified of every build on a Slack channel, and you get an email whenever a build fails.

Jenkins CI with Slack and email notification
Jenkins CI with Slack and Email Notification

The below topics are covered in this blog -

1) Introduction to Jenkins continuous integration
2) Creating a Slack account and incoming webhook
3) Configuring the Slack Notification plugin in Jenkins
4) Configuring email notification in Jenkins
5) Testing the setup

1. Introduction

Jenkins is an open-source continuous integration tool written in Java, used for building, testing, and reporting on isolated changes in a larger codebase in real time. It lets developers find defects and fix them rapidly through automated testing.

2. Create a Slack Account and Incoming Webhook

Slack is a chat platform that provides incoming webhooks, which let external systems post messages into a channel. We will use one so Jenkins can send build notifications into your Slack channel.

If you haven't created your Slack account yet, sign up first, then visit:

https://slack.com/signin?redir=/apps/manage

Or: Channel Administrator -> Manage Apps -> Create Incoming Webhooks

Once you have an account, go to Slack → Administration → Manage Apps.

Slack Manage Apps screen
Slack Manage Apps

Search for Incoming WebHooks:

Searching for Incoming WebHooks in the Slack app directory
Search for Incoming WebHooks

You will get the screen below:

Slack Incoming WebHooks app page
Incoming WebHooks

Click Add to Slack and select the channel you want build notifications posted to:

Selecting the Slack channel for the incoming webhook
Select Channel Name

Click Add Incoming WebHook Integration. You will get a Slack webhook URL that looks like this:

https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX

Treat this URL as a secret. Anyone who has it can post messages into your channel. Never commit it to a repository, paste it in a screenshot, or publish it.

You can test it with a curl command:

curl -X POST --data-urlencode \
  'payload={"channel": "#codeztech-jenkins", "username": "webhookbot",
   "text": "This is posted to #codeztech-jenkins from a bot named webhookbot.",
   "icon_emoji": ":ghost:"}' \
  https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX
Slack incoming webhook URL configuration screen
Slack Webhook URL

3. Configure the Slack Notification Plugin in Jenkins

Make sure the Slack Notification plugin is installed from Manage Jenkins → Manage Plugins. You can download it directly from:

https://updates.jenkins.io/download/plugins/slack/

Then go to Manage Jenkins → Upload Plugin and restart Jenkins. I have made a full video on this, linked at the bottom of this post. Feel free to comment on my YouTube channel or here if you run into any issue.

Once restarted, go to Manage Jenkins → Configure System and scroll to the Slack section.

Base URL        https://hooks.slack.com/services/
Team Subdomain  your Slack workspace subdomain
Credential      the webhook token, stored as a Jenkins secret
Channel         #your-channel-name

You can find the team subdomain in your Slack URL path:

Finding the Slack team subdomain for the incoming webhook
Slack - Incoming WebHook Team Subdomain

Add the credential and enter your channel name prefixed with # — the channel where you want all build notifications sent. Click Test Connection to confirm the setup works.

Configuring Slack notification settings in Jenkins
Configure Slack on Jenkins

4. Configure Email Notification in Jenkins

To configure email notification in Jenkins, go to Manage Jenkins → Configure System → E-mail Notification and use the following settings:

SMTP Server           smtp.gmail.com
Default user e-mail   your-address@gmail.com
SMTP Authentication   checked
Username              your-address@gmail.com
Password              your app password (see below)
Use SSL               checked
SMTP Port             465
Reply-To Address      an alternate address

Do not use your normal Gmail password — Google will reject it. Generate an app password instead:

Step 1 — Sign in to Gmail and go to Manage your Google Account
Step 2 — Open the Security tab and find the sign-in section
Step 3 — Click App passwords and select the device or app
Step 4 — Generate the password and copy it

Paste that app password into the Jenkins password field, then click Test configuration to send a test email.

Configuring email notification settings in Jenkins
Configure Email Notification on Jenkins

5. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Sunday, January 26, 2020

Jenkins: How to Create Builds with the Jenkins Freestyle Project


A Jenkins freestyle project is a repeatable build made up of build steps and post-build actions. You choose what runs at each stage, and there are dozens of standard plugins available to extend it. This blog walks through creating a freestyle job from scratch, pointing it at a Git repository, compiling a Java program, and checking the console output.

Jenkins create and build freestyle project
Jenkins create & build freestyle project

The below topics are covered in this blog -

1) What is a freestyle project in Jenkins?
2) Creating a new freestyle build job
3) Configuring source code management with Git
4) Adding a build step
5) Running the build and reading the console output

1. Freestyle Project in Jenkins

A Jenkins project consists of build steps and post-build actions. The actions you can perform fall into three places: the build step, the post-build action, or a Jenkins pipeline. There are a large number of standard plugins available to cover most common requirements.

2. Creating a Freestyle Build Job

To create a Jenkins freestyle job, log in to your Jenkins dashboard. If Jenkins is hosted locally, visit http://localhost:8080. If it is hosted elsewhere, use that URL instead.

Step 1

Click New Item in the top left corner of your dashboard.

Jenkins New Item menu option on the dashboard
Jenkins New Item

Step 2

Enter the name of the item, choose Freestyle project, and click OK.

Selecting Freestyle project when creating a new Jenkins item
Freestyle Project

Step 3

Enter the details of the project you want to set up.

Entering Jenkins freestyle project details
Jenkins Project Details

Step 4

Under Source Code Management, enter your repository URL so Jenkins can clone it. For this example the URL is:

https://github.com/kriru/firstJava.git
Configuring the Git repository URL in Jenkins source code management
Jenkins Setup Git Project

Step 5

Click Add build step → Execute Windows batch command and add the commands you want to run. In our case:

javac HelloWorld.java
java HelloWorld

On a Linux or macOS agent, use Execute shell instead of Execute Windows batch command — the two commands themselves stay the same.

Adding an Execute Windows batch command build step in Jenkins
Jenkins Build Windows Batch Command

Step 6

Click Apply & Save, then Build Now to check whether the build succeeds.

Triggering a Jenkins build with Build Now
Jenkins Build Now
Jenkins build successful status on the project page
Jenkins Build Successful

Step 7

If you want to explore further, click Console Output to see the detailed result.

Jenkins console output showing the completed build log
Jenkins Console Output

Congratulations — we have executed our HelloWorld program from Jenkins.

3. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Jenkins : How to download and install Jenkins


A Jenkins freestyle project is a repeatable build made up of build steps and post-build actions. You choose what runs at each stage, and there are dozens of standard plugins available to extend it. This blog walks through creating a freestyle job from scratch, pointing it at a Git repository, compiling a Java program, and checking the console output.

Jenkins create and build freestyle project
Jenkins create & build freestyle project

The below topics are covered in this blog -

1) What is a freestyle project in Jenkins?
2) Creating a new freestyle build job
3) Configuring source code management with Git
4) Adding a build step
5) Running the build and reading the console output

1. Freestyle Project in Jenkins

A Jenkins project consists of build steps and post-build actions. The actions you can perform fall into three places: the build step, the post-build action, or a Jenkins pipeline. There are a large number of standard plugins available to cover most common requirements.

2. Creating a Freestyle Build Job

To create a Jenkins freestyle job, log in to your Jenkins dashboard. If Jenkins is hosted locally, visit http://localhost:8080. If it is hosted elsewhere, use that URL instead.

Step 1

Click New Item in the top left corner of your dashboard.

Jenkins New Item menu option on the dashboard
Jenkins New Item

Step 2

Enter the name of the item, choose Freestyle project, and click OK.

Selecting Freestyle project when creating a new Jenkins item
Freestyle Project

Step 3

Enter the details of the project you want to set up.

Entering Jenkins freestyle project details
Jenkins Project Details

Step 4

Under Source Code Management, enter your repository URL so Jenkins can clone it. For this example the URL is:

https://github.com/kriru/firstJava.git
Configuring the Git repository URL in Jenkins source code management
Jenkins Setup Git Project

Step 5

Click Add build step → Execute Windows batch command and add the commands you want to run. In our case:

javac HelloWorld.java
java HelloWorld

On a Linux or macOS agent, use Execute shell instead of Execute Windows batch command — the two commands themselves stay the same.

Adding an Execute Windows batch command build step in Jenkins
Jenkins Build Windows Batch Command

Step 6

Click Apply & Save, then Build Now to check whether the build succeeds.

Triggering a Jenkins build with Build Now
Jenkins Build Now
Jenkins build successful status on the project page
Jenkins Build Successful

Step 7

If you want to explore further, click Console Output to see the detailed result.

Jenkins console output showing the completed build log
Jenkins Console Output

Congratulations — we have executed our HelloWorld program from Jenkins.

3. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Sunday, January 5, 2020

Amazon EC2 Tutorial : Hosting Mean Stack Application


Amazon EC2 is one of the most used and most fundamental services in AWS, so it makes sense to start there when you are new to the platform. But the first question is: what is EC2 in AWS?

Hosting a MEAN stack application on Amazon EC2
Amazon EC2 — Hosting a MEAN Stack Application

The below topics are covered in this blog -

1) What is EC2 in AWS?
2) The MEAN stack CRUD application
3) Architecture of the application
4) Installing MongoDB on an EC2 instance
5) Hosting the MEAN stack with PM2

1. What is EC2 in AWS?

To keep it simple:

  1. EC2 is basically a virtual machine with its own CPU, RAM, and storage.
  2. It lets you choose a platform — Windows, Linux, Ubuntu, and so on — when you launch the machine in the cloud.
  3. Each virtual machine is called an instance, and you can start, stop, or terminate it whenever you like.

2. AWS EC2 MEAN Stack Application Screens

Here are a few screens from the MEAN stack CRUD application we are going to host on AWS EC2.

Insert

MEAN stack CRUD application insert screen
MEAN Stack CRUD Application — Insert

Update

MEAN stack CRUD application update screen
MEAN Stack CRUD Application — Update

Delete

MEAN stack CRUD application delete screen
MEAN Stack CRUD Application — Delete

3. Architecture of the MEAN Stack Application

The diagram below shows the architecture we are setting up across two AWS instances. The frontend uses Angular, and every user interaction — create, read, update, delete — goes through Node.js.

MEAN stack application architecture across two AWS EC2 instances
MEAN Stack Application Architecture

Node.js acts as the mediator that talks to the MongoDB database and returns data to the frontend. We use two instances — one for the frontend and one for the backend.

MEAN stack frontend project:

git clone https://bitbucket.org/atique12/mongocrud_application_frnt_end.git

MEAN stack backend project:

git clone https://bitbucket.org/atique12/mongocrud_application_back_end.git

If you want to learn more about the MEAN stack, see my related blog post: MEAN Stack Angular CRUD Application.

4. Installing MongoDB on an AWS EC2 Instance

If you haven't registered on AWS yet, do that first. Then launch an Ubuntu instance, download the PEM key, and open your terminal:

ssh -i yourkey.pem ubuntu@your_ec2_public_ip

Once you are logged in, install MongoDB. Reference links:

MongoDB — Install MongoDB Community Edition
GUI client — Robo 3T (formerly Robomongo)

Reference repository for this tutorial:

git clone https://bitbucket.org/atique1224/aws-ec2-tutorial.git

5. Hosting the MEAN Stack with PM2

On the second Ubuntu instance, install Node.js, NVM, the Angular CLI, and PM2, then clone the code onto the instance.

Install Node.js

sudo apt update
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs

Install NVM (to switch Node.js versions)

wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
source ~/.profile
nvm ls-remote

Install the Angular CLI

sudo npm install -g @angular/cli

Install PM2 (to keep the app running)

sudo npm install -g pm2

Clone both projects

git clone https://bitbucket.org/atique12/mongocrud_application_frnt_end.git
git clone https://bitbucket.org/atique12/mongocrud_application_back_end.git

Point the code at your instances. By default the projects use localhost. You need to replace that with your EC2 public IPs:

  • Backend — in db.js, set the MongoDB instance's public IP; in index.js, set the backend instance's public IP.
  • Frontend — in src/app/shared/person.services.ts, change localhost to the backend instance's public IP:
vi src/app/shared/person.services.ts

One important note: pointing the frontend at a raw EC2 public IP hardcodes an address that changes every time the instance restarts, unless you attach an Elastic IP. For anything beyond a demo, put the backend behind a stable hostname or load balancer rather than baking the IP into your source.

6. Video Walkthrough

More videos in this series:


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Mean Stack Angular Crud Application Tutorial


The term MEAN stack refers to a collection of JavaScript-based technologies used to build web applications. MEAN stands for MongoDB, Express.js, Angular, and Node.js.

MEAN stack application - MongoDB, Express, Angular, Node.js
MEAN Stack Application

The below topics are covered in this blog -

1) Overview of the MEAN stack
2) What does CRUD stand for?
3) Architecture of the MEAN stack application
4) Building the Node.js backend
5) Building the Angular frontend

1. Overview of the MEAN Stack

MongoDB — MongoDB is a schema-less NoSQL database. It stores data in a binary JSON (BSON) format, which makes it easy to pass data between client and server. If you haven't seen my MongoDB tutorial yet, see my related post: Complete Guide to MongoDB.

Angular — Angular is a JavaScript framework developed by Google. It provides features like two-way binding, routing, sharing data between components, RxJS, and observables. In short, it is a complete solution for rapid frontend development.

Node.js — Node.js is a server-side JavaScript execution environment built on Google Chrome's V8 runtime. It helps you build highly scalable, concurrent applications quickly. See my related post: Complete Guide to Build a RESTful API with Node.js.

Express.js — Express is a lightweight framework for building web applications in Node.js. It provides a robust set of features for single- and multi-page apps and was inspired by the Ruby framework Sinatra. We use Express in Node.js as our web application framework.

2. What Does CRUD Stand For?

CRUD stands for Create, Read, Update, and Delete — the four basic operations of persistent storage.

CRUD stands for Create, Read, Update, Delete
CRUD Stands For

We are going to build an application that performs all four operations using the MEAN stack.

3. Architecture of the MEAN Stack Application

The diagram below shows the architecture of the application we are building. The frontend uses Angular, and every user interaction — create, read, update, delete — goes through Node.js.

MEAN stack application architecture with Angular, Node.js and MongoDB
MEAN Stack Application Architecture

Node.js acts as the mediator that talks to the MongoDB database and returns data to the frontend.

4. Building the Node.js Backend

Step 1 — Generate a package.json. Create a folder, then run:

mkdir meanstack_backend
cd meanstack_backend
npm init          # press enter through the prompts
npm install body-parser cors express mongoose nodemon --save

Step 2 — Create the entry file, index.js:

const express = require('express');
const bodyParser = require('body-parser');
const cors = require('cors');

const mongoose = require('./db.js');
const personController = require('./controllers/personController.js');

var app = express();
app.use(bodyParser.json());
app.use(cors({ origin: 'http://localhost:4200' }));

// Application testing API
app.get('/app/testing', (req, res) => {
  res.send({
    "message": "App is working fine",
    "status": 200
  });
});

app.use('/person', personController);

app.listen(3000, () => console.log('Server connected to port: 3000'));

db.js — the database connection:

const mongoose = require('mongoose');

mongoose.connect('mongodb://localhost:27017/firstdb', {
  useNewUrlParser: true,
  useUnifiedTopology: true
}, (err) => {
  if (!err) {
    console.log('MongoDB connected ....');
  } else {
    console.log('Error in DB connection: ' + JSON.stringify(err, undefined, 2));
  }
});

module.exports = mongoose;

model/person.js — the data model:

const mongoose = require('mongoose');

var Person = mongoose.model('Person', {
  name:  { type: String },
  mail:  { type: String },
  class: { type: Number }
});

module.exports = { Person };

controllers/personController.js — the CRUD routes:

const express = require('express');
var router = express.Router();

var { Person } = require('../model/person.js');
var ObjectId = require('mongoose').Types.ObjectId;

// Get all users
router.get('/', (req, res) => {
  Person.find((err, doc) => {
    if (!err) { res.send(doc); }
    else { console.log('Error retrieving persons: ' + JSON.stringify(err, undefined, 2)); }
  });
});

// Insert a user
router.post('/', (req, res) => {
  var per = new Person({
    name:  req.body.name,
    mail:  req.body.mail,
    class: req.body.class
  });

  Person.findOne({ 'mail': req.body.mail }, (err, docs) => {
    if (!docs) {
      per.save((err, doc) => {
        if (!err) {
          res.status(200).send({ auth: true, doc, message: '1 document inserted' });
        } else {
          console.log('Error inserting data: ' + JSON.stringify(err, undefined, 2));
        }
      });
    } else {
      console.log('User already exists: ' + req.body.mail);
      res.status(400).send({ message: 'User already exists: ' + req.body.mail });
    }
  });
});

// Update a user
router.put('/:id', (req, res) => {
  if (!ObjectId.isValid(req.params.id))
    return res.status(400).send(`No record with given id: ${req.params.id}`);

  var per = {
    $set: {
      name:  req.body.name,
      mail:  req.body.mail,
      class: req.body.class
    }
  };

  Person.findOneAndUpdate({ mail: req.body.mail }, per, { new: true, useFindAndModify: false }, (err, doc) => {
    if (err) {
      console.log('Something went wrong when updating data.');
    } else if (doc) {
      res.status(200).send({ auth: true, message: '1 document updated' });
    } else {
      res.status(404).send({ message: 'Resource not found. Please register first.' });
    }
  });
});

// Delete a user
router.delete('/:id', (req, res) => {
  if (!ObjectId.isValid(req.params.id))
    return res.status(400).send(`No record with given id: ${req.params.id}`);

  Person.findByIdAndRemove(req.params.id, { useFindAndModify: false }, (err, docs) => {
    if (!err) { res.send(docs); }
    else { console.log('Error deleting person: ' + JSON.stringify(err, undefined, 2)); }
  });
});

module.exports = router;

You can also clone the finished backend:

git clone https://bitbucket.org/atique12/mongocrud_application_back_end.git

5. Building the Frontend with Angular

The frontend uses Angular with Angular Material for the design. You can clone the finished frontend:

git clone https://bitbucket.org/atique12/mongocrud_application_frnt_end.git

6. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Thursday, December 26, 2019

Logstash Tutorial : A complete guide for the beginners how to index data from logstash to Elasticsearch and Kibana


Logstash is an open-source, server-side data processing pipeline that ingests data from many sources at once, transforms it, and then sends it to your favourite "stash" — most commonly Elasticsearch.

Logstash data processing pipeline logo
Logstash

The below topics are covered in this blog -

1) Overview of Logstash
2) What is Logstash?
3) Installing Logstash
4) Downloading a sample dataset
5) How to run Logstash
6) A complete simple.conf example

1. Overview of Logstash

Developed by: Elastic NV. Logstash is one of the three core components of the Elastic (ELK) stack, alongside Elasticsearch and Kibana.

2. What is Logstash?

Logstash is a lightweight, open-source, server-side data processing pipeline that lets you collect data from a variety of sources, transform it on the fly, and send it to your chosen destination. It is most often used as the data pipeline for Elasticsearch. Thanks to its tight Elasticsearch integration, powerful log-processing capabilities, and a large ecosystem of pre-built plugins, Logstash is a popular choice for loading data into Elasticsearch.

Logstash pipeline diagram showing inputs, filters and outputs
The Logstash Pipeline — Inputs, Filters, Outputs

3. Installing Logstash

Download Logstash from the official Elastic site. Always match the Logstash version to your Elasticsearch version:

https://www.elastic.co/downloads/logstash

4. Download a Dataset to Import into Elasticsearch

You can grab a free sample dataset from Kaggle. This tutorial uses the IBM HR Analytics employee attrition dataset:

https://www.kaggle.com/datasets

5. How to Run Logstash

A Logstash pipeline has three stages: input (where the data comes from), filter (how it is parsed and transformed), and output (where it goes). At its simplest:

input {
  stdin { }
}

filter { }

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Save the file with a .conf extension, then run it from your Logstash folder:

bin/logstash -f simple.conf     # simple.conf is your config file name

Official configuration reference: Logstash configuration guide.

6. A Complete simple.conf Example

This config reads a CSV, converts the numeric columns to integers, and indexes the result into Elasticsearch. Update the path to point at your own CSV file.

input {
  file {
    path => "/path/to/your/employee.csv"
    start_position => "beginning"
    sincedb_path => "NUL"
  }
}

filter {
  csv {
    separator => ","
    columns => [ "Age","Attrition","BusinessTravel","DailyRate","Department",
      "DistanceFromHome","Education","EducationField","EmployeeCount",
      "EmployeeNumber","EnvironmentSatisfaction","Gender","HourlyRate",
      "JobInvolvement","JobLevel","JobRole","JobSatisfaction","MaritalStatus",
      "MonthlyIncome","MonthlyRate","NumCompaniesWorked","Over18",
      "OverTime","PercentSalaryHike","PerformanceRating",
      "RelationshipSatisfaction","StandardHours","StockOptionLevel",
      "TotalWorkingYears","TrainingTimesLastYear","WorkLifeBalance",
      "YearsAtCompany","YearsInCurrentRole","YearsSinceLastPromotion",
      "YearsWithCurrManager" ]
  }

  mutate {
    convert => {
      "Age" => "integer"
      "DailyRate" => "integer"
      "DistanceFromHome" => "integer"
      "Education" => "integer"
      "EmployeeCount" => "integer"
      "EmployeeNumber" => "integer"
      "EnvironmentSatisfaction" => "integer"
      "HourlyRate" => "integer"
      "JobInvolvement" => "integer"
      "JobLevel" => "integer"
      "JobSatisfaction" => "integer"
      "MonthlyIncome" => "integer"
      "MonthlyRate" => "integer"
      "NumCompaniesWorked" => "integer"
      "PercentSalaryHike" => "integer"
      "PerformanceRating" => "integer"
      "RelationshipSatisfaction" => "integer"
      "StandardHours" => "integer"
      "StockOptionLevel" => "integer"
      "TotalWorkingYears" => "integer"
      "TrainingTimesLastYear" => "integer"
      "WorkLifeBalance" => "integer"
      "YearsAtCompany" => "integer"
      "YearsInCurrentRole" => "integer"
      "YearsSinceLastPromotion" => "integer"
      "YearsWithCurrManager" => "integer"
    }
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "employee"
  }
  stdout { }
}

Clone the full project:

git clone https://bitbucket.org/atique1224/youtube_logstash_tutorial.git

7. Video Walkthrough


About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Kibana Tutorial : A Complete guide for the beginners


Kibana is an open-source data visualisation dashboard for Elasticsearch. It provides visualisation on top of the content indexed in an Elasticsearch cluster — users can build bar, line, and scatter plots, pie charts, and maps over large volumes of data.

Kibana data visualisation dashboard for Elasticsearch
Kibana

The below topics are covered in this blog -

1) Overview of Kibana
2) What is Kibana?
3) Installing Kibana
4) Indexing data into Kibana
5) GET, POST, PUT, DELETE from Kibana Dev Tools
6) Elasticsearch aggregation and projection
7) Elasticsearch pagination and scroll
8) Visualising Elasticsearch data in Kibana

1. Overview of Kibana

Kibana also includes a presentation tool called Canvas, which lets users build slide decks that pull live data directly from Elasticsearch. The combination of Elasticsearch, Kibana, and Logstash is known as the ELK Stack.

Developed by: Elastic NV.

2. What is Kibana?

Kibana is a data visualisation and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, maps, and other diagrams.

3. Installing Kibana

Download Kibana from the official Elastic site. Always match the Kibana version to your Elasticsearch version:

https://www.elastic.co/downloads/kibana

4. Indexing Data into Kibana

When you index data into Elasticsearch, run Kibana alongside it so the data becomes available to both. Clone the project:

git clone https://bitbucket.org/atique1224/youtube_elasticsearch_indexing_tutorial.git

Reference tools used in this tutorial: Node.js, Visual Studio Code, MongoDB, and Robo 3T.

5. GET, POST, PUT, DELETE from Kibana Dev Tools

Kibana's Dev Tools console lets you run REST requests against Elasticsearch directly. Clone the project:

git clone https://bitbucket.org/atique1224/youtube_kibana_tutorial.git

6. Elasticsearch Aggregation and Projection

Aggregation lets you compute summaries over your data — sums, counts, and grouped buckets.

# Aggregation with a sum query
GET /students/_search
{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "Casual_Leaves": {
      "sum": { "field": "leaves.CL" }
    }
  }
}

# Aggregation with a terms bucket (count of repeated values)
GET /students/_search
{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "count": {
      "terms": {
        "field": "dept.keyword",
        "size": 100,
        "order": { "_key": "desc" }
      }
    }
  }
}

# Total count of records
GET /students/_count
{
}

Projection returns only the fields you ask for, using _source:

# Return only selected fields
GET /students/_search
{
  "_source": ["student_id", "skills"],
  "query": {
    "match_all": {}
  }
}

7. Elasticsearch Pagination and Scroll

Pagination. Suppose you have a huge index — ten million records or more. Returning them all to the frontend in one shot is not possible, so Elasticsearch offers two solutions. The first is pagination, using from and size to fetch one page at a time.

# Page through results: 10 at a time, starting at record 20
GET /students/_search
{
  "from": 20,
  "size": 10,
  "query": {
    "match_all": {}
  }
}

Scroll. The second solution is the scroll API. Each request creates a scroll ID and you set an expiry time, after which that scroll ID expires. It is designed for deep, sequential retrieval of large result sets rather than user-facing paging.

# Open a scroll that stays valid for 1 minute
GET /students/_search?scroll=1m
{
  "size": 1000,
  "query": {
    "match_all": {}
  }
}

# Fetch the next batch using the returned _scroll_id
GET /_search/scroll
{
  "scroll": "1m",
  "scroll_id": "PASTE_YOUR_SCROLL_ID_HERE"
}

Note: for deep pagination in current Elasticsearch, the search_after parameter with a point-in-time (PIT) is now the recommended approach over scroll for most use cases. Scroll still works but is considered legacy for real-time paging.

8. Visualising Elasticsearch Data in Kibana

To visualise Elasticsearch data, the data must first be present in Kibana, and you need to create an index pattern for the relevant indices. Good visualisations also depend on good data — the "4 Cs" of data quality:

  • Correctness — validate data accuracy against an external reference.
  • Currency — deliver new and updated content in a timely manner.
  • Completeness — provide the right attributes and analysis so users have all the information they need to make decisions.
  • Consistency — standardise identifiers and content across databases and products, so users get consistent information regardless of platform.

About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub

Elasticsearch Tutorial : A Complete guide for the beginners


Elasticsearch is a highly scalable, open-source, full-text search and analytics engine. It lets you store, search, and analyse large volumes of data quickly and in near real time. It is generally used as the underlying engine that powers applications with complex search requirements. Elasticsearch builds a distributed system on top of Apache Lucene for indexing and automatic type guessing, and exposes Lucene's features through a JSON-based REST API.

Elasticsearch search and analytics engine logo
Elasticsearch

The below topics are covered in this blog -

1) Overview of Elasticsearch
2) SQL vs NoSQL
3) Relational vs non-relational databases
4) Scale up vs scale out
5) What is Elasticsearch?
6) How does Elasticsearch work?
7) Real-time example — Case 1
8) Real-time example — Case 2
9) Companies using Elasticsearch
10) Installing Elasticsearch
11) Bulk indexing from MongoDB to Elasticsearch
12) Elasticsearch with Node.js

1. Overview of Elasticsearch

Developed by: Elastic NV, first released on 8 February 2010.

Features:

  • Data storage and a document store for unstructured data
  • Flexible data types and full-text search
  • Field- and document-level APIs
  • Cluster indices, data snapshots, and rollup indices
  • Elasticsearch SQL and role-based access control

2. SQL vs NoSQL

Before going deeper into Elasticsearch, it helps to understand SQL and NoSQL, because the distinction is central to how Elasticsearch scales.

  • SQL databases scale vertically — to grow, you increase the capacity (CPU, RAM) of a single server.
  • NoSQL databases scale horizontally — to grow, you add more servers.

3. Relational vs Non-Relational Databases

In a relational (scale-up) database you have primary keys, secondary keys, and foreign keys, along with joins — left outer join, right outer join, full join, and so on. As the data grows on a single server, you increase that server's RAM and CPU.

In a non-relational (scale-out) database there is no join concept. Instead of increasing the power of one server, you add more servers to share the load.

4. Scale Up vs Scale Out

Why cover all this in a post about Elasticsearch? Because Elasticsearch is a NoSQL, scale-out database, and understanding scale-out is what makes the rest of it make sense. Rather than growing one machine, Elasticsearch distributes data and load across many nodes.

5. What is Elasticsearch?

Elasticsearch is an open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release it has become one of the most popular search engines, commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence.

6. How Does Elasticsearch Work?

Raw data flows into Elasticsearch from a variety of sources — logs, system metrics, web applications, and more. During ingestion, this raw data is parsed, normalised, and enriched before being indexed. Once indexed, you can run complex queries against it and use aggregations to retrieve rich summaries. From Kibana, you can then build visualisations, share dashboards, and manage the Elastic Stack.

7. Real-Time Example — Case 1

Consider the basic architecture of a web application dealing with a huge amount of data. The frontend is a web browser. When a user searches, if a large volume of data sits in a traditional database, it becomes difficult to return relevant results quickly. This is exactly where Elasticsearch comes in — it sits alongside the primary database and serves fast search results.

Web application architecture using Elasticsearch for search
Real-Time Example — Case 1

8. Real-Time Example — Case 2

In the second scenario, when you have a large amount of data to work with and surface to the frontend, Elasticsearch again fits well. And once your data is in Elasticsearch, you can visualise it through Kibana as a pie chart, bar chart, table, and more.

Elasticsearch with Kibana visualisation architecture
Real-Time Example — Case 2

These are just two examples. There are countless reasons and challenges you will face when dealing with big data — and those help you decide when and where Elasticsearch and Kibana fit into the picture.

9. Companies Using Elasticsearch

Many well-known companies use Elasticsearch, Kibana, Logstash, and Filebeat, including Cisco, SAP, IBM, Citrix, Meta (Facebook), LinkedIn, Microsoft, Red Hat, Adobe, EA, Bosch, eBay, Flipkart, and others.

Is it free? Yes — Elasticsearch is free for many features and services under its open-source and free tiers, with paid options for advanced capabilities.

10. Installing Elasticsearch

Download Elasticsearch from the official Elastic site. Match the version to your Kibana and Logstash versions:

https://www.elastic.co/downloads/elasticsearch

11. Bulk Indexing from MongoDB to Elasticsearch

This section indexes bulk data from MongoDB into Elasticsearch. Clone the project:

git clone https://bitbucket.org/atique1224/youtube_elasticsearch_indexing_tutorial.git

Reference tools: Node.js, Visual Studio Code, MongoDB, and Robo 3T.

12. Elasticsearch with Node.js — Aggregation and CRUD

This section covers using Elasticsearch from Node.js: aggregations and the GET, POST, PUT, and DELETE methods. Clone the project:

git clone https://bitbucket.org/atique1224/youtube_elasticsearch_with_node_js_tutorial.git

About the Author
Atique Ahmed — Principal AI Architect. 7x Microsoft MVP and Guinness World Record holder for Programming Excellence. Founder of Codez Tech.
Portfolio  |  LinkedIn  |  GitHub