How to check the memory utilization of cluster nodes in a Kubernetes Cluster ?

 The memory and CPU utilization of a Kubernetes cluster can be checked by using the following command.

kubectl top nodes 

The above command will list all the nodes in the Kubernetes cluster and shows the CPU utilization and memory utilization.

To list all the nodes in a Kubernetes cluster, use the following command.

kubectl get nodes

How to make a Dice Rolling Simulator using Python ? [ Source code included ]

 Snake and Ladder is a very popular game. We all are familiar with the dice that we use in this game. Here I am explaining a simple program that simulates the dice. I will be using random() function in python to implement this feature. 

 

The random module has so many functionalities and features. Here we will be using the random integer generation feature of the random module.

First lets get the simple program ready. 

import random
dice_value = random.randint(1,6)
print(dice_value)

The random.randint() function is the core logic in this function. The arguments 1 and 6 are the lower and upper limits of the random integer values. The function will return values between both these numbers including these numbers. 

Now let us try to give some more look and feel to the program. I have added some more lines of code to improve usability. This will give a visual effect also.

If you want a more advanced dice simulator, let us modify the program further to with a Graphical User Interface. We will use TKinter package in python to implement this simple user interface.

How to calculate the area of a triangle using a python program ?

 Calculating the area of a triangle is a very fundamental problem statement. Here I am explaining the calculation of the area of a triangle using Heron's Formula.

What is Heron's Formula ?

This is a formula which has been in use from the past 2000 year. We can easily calculate the area of a triangle if we know the measurement of all the three sides of a triangle.

 As shows in the picture above, if we know all the sides of the triangle a, b and c. We will be able to apply Heron's formula.

Area = (s(s-a) * (s-b) * (s-c)) ^ 0.5

where s = (a+b+c) / 2

We will use the above formula in a python program and make a utility to calculate the area in a programmatic way. 

The sample python program to calculate the area of triangle in given below

How to convert a pandas dataframe into list of dictionaries ?

 Pandas is a high-level data manipulation tool commonly used in python. It is build on Numpy and the key data structure is DataFrame.

Here I am explaining a sample python program to convert a Pandas Dataframe into a list of dictionaries.

Program to convert Dataframe into list of dictionaries

In the below program, we are reading a csv file and loading it in pandas. You can configure the delimiter as per your requirement. The required_fields is an optional parameter that can be used to limit or select the required fields in the Dataframe from the csv. By default, it picks up all the fields in the file.

Python Program to Split a large list into list of smaller lists

There are requirements where we have to split a large list into a set of smaller lists. This is a common requirement in places where we use parallel processing. Here is a sample program in python to split a larger list into a set of smaller lists.

 

Simple LED blinking circuit using Raspberry Pi and Python

Introduction

Raspberry Pi is one of the powerful device invented in this era. During my school days and college days, all the automation circuits and hobby circuits were developed  either directly by soldering components in the generic circuit board or by using a micro controllers like Arduino. The set up and debug effort was more and the programming was limited to Arduino processing language.

With the invention of Raspberry Pi, the possibilities are endless. It is a mini computer with all the facilities of a desktop PC. Users can develop and run any programs as they wish. The device also comes with options to connect to several peripherals and extension circuit boards.

Here I will be explaining a very simple program, like the baby step towards the world of automation using Raspberry Pi. We will use Raspberry Pi to control an LED (Turn OFF and ON).

Requirements

•  Raspberry Pi
• SD Card with Raspbian OS
• Bread Board
• Jumper wires
• LED (2 nos)
  

GPIO Pins in Raspberry Pi 

A powerful feature of the Raspberry Pi is the row of GPIO (general-purpose input/output) pins along the top edge of the board. Any of the GPIO pins can be designated (in software) as an input or output pin and used for a wide range of purposes. In this example, we are using a python program to control the GPIO pins and turn ON/OFF LED.

Connection

Connect negative leg of both LED to PIN 6(Ground), one positive leg to PIN 8 and another positive LEG to PIN 10

Program

Raspbian OS will be having Thonny Python IDE by default, so it’s easy to write and execute python codes in Raspberry Pi 

 

Install the RPi.GPIO package in the Raspberry Pi. 

 

       
       pip install RPi.GPIO

 

Copy paste the below program in the IDE. Make the connections between LED and GPIO pins as explained in the previous step.

 

https://gist.github.com/coderinboots/526c64d84b07e1a50a7953d43c888b1e

Execute the program. You will see the lights blinking.

 

Anaconda Vs Miniconda Vs Vanilla Python - Which one to use ? Confused ?

Have you heard about Anaconda or Miniconda ?

If you are a python developer or someone who is interested to learn python, you should definitely know this.

Here I will be explaining these from an individual user's perspective.

Anaconda - This is one of the world's popular opensource python distribution with lot of machine learning and datascience packages. This is the best option for someone who is interested to learn datascience & machine learning. This is like a complete ecosystem of libraries and packages for a data scientist or an advanced python programmer.

Miniconda - This is a minimal version of anaconda.  It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.

Both Anaconda and Miniconda comes with conda package manager. Using conda, we can install the required package from several thousands of cloud repositories. So it is very easy to find and install dependent packages using conda.

Official Python - This is the standard version of Python. This uses pip package manager.

I would recommend miniconda for using as the development environment in windows for basic level python developers as it will be easy to handle the dependencies. Some of the packages will cause trouble while installing using pip in windows.

How to run python applications as system service in CentOS 7 or RHEL 7 ?

I started developing python applications from 2012. During my learning period, I was using screen in linux for running my python applications in the background. Majority of my colleagues were also following the same approach for running applications in background. The advantage of this approach was its easiness. But these screens will go away if the system gets rebooted. Also these applications will never get started on boot.

Systemd is a system and service manager for CentOS 7 and RHEL 7. This the first process that gets started after the system boots. This is compatible with the SysV init scripts used by CentOS 6 and RHEL 6.

It is very easy to run any custom application as a system service. In this way, the management will become very easy.

Suppose I have a python flask application app.py and I want to run it as a system service. I want to run this application under my unix user account amal.  

The contents of app.py is given below.




My code is located in the location /home/amal. Usually in case of any production deployment, the code will not be kept in the user home directory. This is to explain the scenario of running the application as a non root user.

The command to run my application is

python app.py 

For running in production mode with a WSGI like gunicorn, the command will be

gunicorn app:app --bind 0.0.0.0:8080 -w 2

For running this application as system service. We just need to do the following steps.

Create a file myapp.service in the location /etc/systemd/system/ The content of myapp.service is given below.


Now save the file and execute the following command. The following command will reload all the systemd configurations.

systemctl daemon-reload

Now we can start the application with the following command

systemctl start myapp

Now check the status of the application with the following command

systemctl status myapp

Also we can invoke the webservice using curl command.

curl -X GET http://localhost:8080/

How to check the CPU and Memory utilization of a Linux system ?

One of the best and easiest way to perform the realtime CPU and Memory utilization  of a Linux system is by using htop command.

The command will provide an interactive user interface with the summary of load on the system and the details of individual process running on the system. This is very helpful to identify the resource utilization of the applications running in a system.

How to list all the indices in Elasticsearch ?

For listing all the indices from unix command line, type the following command.

curl -X GET http://eshost:9200/_cat/indices

For checking this from the browser, simply type the following URL in the browser

http://eshost:9200/_cat/indices

Where eshost is the hostname of the elasticsearch server. You can replace it with the IP address of the server also. 9200 is the default port used by elasticsearch

What is Inode in Linux ? How to check the Inode usage in Linux ?

The inode is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory. Each inode stores the attributes and disk block location(s) of the object's data. File-system object attributes may include metadata (times of last change, access, modification), as well as owner and permission data. Inode is also known as Index node.

Directories are lists of names assigned to inodes. A directory contains an entry for itself, its parent, and each of its children. Sometimes, we may face space issue in disks which has storage space. Mostly this issue occurs because of inode filling. The file system might have storage space available, but the inode table does not have space.

The command to check the storage space is given below.

df -h

The command to check the inode usage is given below.

df -i

If the inode capacity of a disk is full, we will not be able to write any data to the disk. Usually this issue happens when we keep large number of small files.

How to list all the local users in a linux system ??

The following command can be used to list all the local users in linux.

cut -d: -f1 /etc/passwd

This will list all the local users in the system. This will not list all the users if it is integrated with LDAP or any other similar systems

How to import data from RDBMS to Hadoop and viceversa ?

Hadoop became very popular within few years because of its robust design, open source and ability to handle large data. Nowadays lot of RDBMS to hadoop migration projects are happening. Hadoop is not a replacement for the RDBMS, but for certain usecase, hadoop can perform well than RDBMS. Some projects may require data from rdbms along with multiple sources for finding insights. In these scenarios, we need to transfer data from RDBMS to hadoop environment. This task sounds simple, but this is a difficult task as this involves lot of risk. The possible solutions for importing data from RDBMS to hadoop are explained below

1) Using SQOOP
Sqoop is a hadoop ecosystem component that is developed for importing data from RDBMS to hadoop and for exporting data from hadoop to RDBMS. Sqoop jobs runs as a mapreduce job. Sqoop utilizes hadoop's parallelism for doing the parallel import and export. Internally sqoop is running as a mapper alone job that utilizes jdbc. For using sqoop, we need a good network connectivity between the RDBMS environment and hadoop environment.

2) By dumping the data from database and transferring via portable secondary storage devices
Most of the companies may not allow direct network connectivity to RDBMS environment from hadoop. Another reason for not allowing is that when a sqoop job is triggered the data flow through the network will be very high which will affect the performance of other systems connected to the network. In such cases, data will be transferred to the hadoop environment by dumping the data from the database, copying it to some portable secondary storage devices or some cloud storage (if allowed) and transferring the data to hadoop environment.

How to add new nodes to a hadoop cluster ?

Hadoop doesn't require any downtime for adding new nodes to the cluster. The following steps explains the procedure for adding new nodes to an existing hadoop cluster.

• Get the machine ready with proper hardware and compatible software.
• Install datanode & tasktracker/nodemanager in the new machine. 
• Configure the data storage locations and temporary storage locations properly.
• Add proper configuration files same as that of existing nodes in the hadoop cluster.
• Add the new machine to the network properly.
• Start the datanode & tasktracker/nodemanager services in the new machines.
• The new machines will be automatically added to the hadoop cluster. You can verify this from the namenode and jobtracker/resource manager UI.
• If the hadoop cluster is configured with allowed hosts, add the hostnames of new machines to the allowed hosts file.
• After the nodes are added to the cluster you can run the balancer if required

Map and reduce phase internals - Passing Key and Value pairs from mapper to reducer

Suppose the mappers generated the following output

(coder,1)
(fox,1)
(in,1)
(boots,1)
(coder,1)
(hadoop,1)

How many keys will be passed to the reducers reduce() method ?

Ans : five

Reason: The input to the reduce method of reducer will be key and list of values. The output of the mapper will be grouped based on the keys and will be send to the reducer. Here in this case after grouping based on keys we will get an output like
coder,[1,1]
fox,[1]
in,[1]
boots,[1]
hadoop,[1]

Here we have five unique keys, so the five keys will be sent to the reducer.

Role of partitioner in mapreduce execution

Mapreduce job has mainly two phases, map phase and reduce phase. The output of the mapper will be passed to the reducer. The input and output of mapper will be key-value pair. The input of reducer will be key and list of values. The input to the reducer are the portion of the map output or the complete map output. The map output will be partitioned based on keys and multiple partitions will be generated. Each partition will be send to reducers. The output of map is partitioned in such a way that all the similar keys will be grouped in a single partition and will be send to the same reducer. Otherwise we will get incorrect output. The default partitioner in hadoop uses the hash value of the keys for partitioning the map outputs. This can be overridden by writing user defined partitioning function.

When will the reduce tasks start execution in a mapreduce job.?

Reduce tasks will be started only after the completion of all map tasks. Sometimes we may see the progression of reduce tasks as 15% and the map tasks as 90%. This doesn't means that the reduce tasks started execution. The reduce logic has to be executed on the complete output of all the mappers, not on the partial output. 

The reduce() method will not be called until all of the mappers have completed. If the reduce logic is applied on the partial map output, we will get incorrect output. The 15% or some value pf progression of reduce tasks before the completion of map phase is not the reduce logic execution progress, it is the progress representing the transfer of mapper output to the reducers. The mapred.reduce.slowstart.completed.maps property specifies the percentage of mappers that must complete before the reducers can start receiving data from the completed mappers. Once all the mapper execution gets completed, the reducer starts executing on to of the map outputs.

What happens to a mapreduce job with a reduce class when we set the number of reduce tasks as zero ?

When we set the number of reduce tasks as zero, reduce tasks will not be executed. The output of the mapper will be copied to the hdfs and it will be the output of the job. Suppose 10 mappers were spawned for a job, if we set the number of reduce tasks as zero, we will get 10 output files.
The output files will be with a name similar to part-m-00000, part-m-00001 ..... part-m-00009.
We can set the number of reduce tasks as zero either from the program or from the commandline.

In the program we can set this by setting the following configuration
job.setNumReduceTasks(0);

From the  command line also we can achieve the same result by using the property below
-Dmapred.reduce.tasks=0

What happens to a mapreduce job if the user sets the number of reduce tasks as one ?

When the number of reduce tasks is set to one, only one reduce task will be executed for the entire jobs. All the intermediate map outputs will be gathered by a single reducer. The single reducer processes the entire map outputs and the output will be stored in a single file in hdfs. It will be with the name part-r-00000.
For setting the number of reduce tasks as one, add the following property in the driver class.
job.setNumReduceTasks(1);

What is the best tool for creating workflow or chaining jobs in hadoop ?

Sometimes we require some tools to chain mapreduce jobs, hive jobs, pig jobs etc. We can chain these jobs using our own way either by using programs or using some scripts. But the best way to chain jobs in hadoop ecosystem is by using oozie.

Oozie is a workflow and orchestration framework in hadoop ecosystem. We don't need to worry about the complexities of handling various scenarios that may have to be considered while developing a chaining tool. Oozie is a very simple tool and the workflows can be achieved by using an xml file. For more details refer oozie website