Home Uncategorized Providing Elasticity to Data Node Storage in Hadoop through LVM

Providing Elasticity to Data Node Storage in Hadoop through LVM

by admin

Set up Master Node :

Configuring /etc/hadoop/hdfs-site.xml file : 

Stop the firewall temporarily :

systemctl stop firewalld

Make a directory for namenode :

mkdir /namenode

Format the namenode directory :

hadoop namenode -format

Start the namenode : start namenode

Namenode has been started successfully :

Set up Data Node :

Configuring /etc/hadoop/hdfs-site.xml file :

Configuring /etc/hadoop/core-site.xml file :

Make a directory for Data Node :

mkdir /datanode

Start the Data Node services : start datanode

Data Node started successfully :

Check the hadoop cluster report :

hadoop dfsadmin -report

If you notice, entire hard disk has been allocated to the Hadoop Name Node!!

Integrating LVM with the Data Node Storage Directory :

Now, I’ve attached 2 new hard disks of sizes 5GiB and 4GiB. Run the below command to view the hard disks :

fdisk -l

In my VM, the names of hard disks are “nvme0n2” and “nvme0n3”.
First step is to create physical volumes from hard disks :
pvcreate /dev/nvme0n2
pvcreate /dev/nvme0n3

Check if the physical volumes are created or not :
pvdisplay /dev/nvme0n2
pvdisplay /dev/nvme0n3

Next step is to create a Volume Group for the two physical volumes created :
vgcreate hadoop_VG /dev/nvme0n2 /dev/nvme0n3

Check if the volume group is created or not :
vgdisplay hadoop_VG

Next, create Logical Volume from the Volume Group. Here I’m creating a Logical Volume of 6GiB :
lvcreate --size 6G --name hadoop_LV hadoop_VG

Check if the Logical Volume is created or not :
lvdisplay hadoop_VG/hadoop_LV

Now format the Logical Volume created above :
mkfs.ext4 /dev/hadoop_VG/hadoop_LV

Now, create a directory and mount the logical volume to that directory :
mkdir /dir
mount /dev/hadoop_VG/hadoop_LV
Check if the LV is mounted or not with the command “df -h” :

Now, update the directory name inside /etc/hadoop/hdfs-site.xml file of Data Node :

Stop and then Start the Data Node : stop datanode start datanode

Check the hadoop report :
hadoop dfsadmin -report

See, the DataNode is contributing only ~ 6GiB Storage to the cluster !!

Increasing the size of Logical Volume :
Command to increase the size of Logical Volume by 2GiB :
lvextend --size +2G /dev/hadoop_VG/hadoop_LV

We have to format the newly extended 2GiB storage with the command :
resize2fs /dev/hadoop_VG/hadoop_LV

Check the hadoop cluster report. The size of storage will be increased :

Let’s upload a file to the cluster :

You can view the file in web view of hadoop :

Decreasing the size of Logical Volume :
Stop the datanode : stop datanode
First unmount the logical volume :
umount /dir
Clean and scan the logical volume :
e2fsck -f /dev/hadoop_VG/hadoop_LV

Recreate Inode table upto the required size. I’m decreasing it to 3GiB :
resize2fs /dev/hadoop_VG/hadoop_LV 3G

Now reduce the size of Logical Volume with lvreduce command :
lvreduce --size 3G /dev/hadoop_VG/hadoop_LV

Now, mount the LV :
mount /dev/hadoop_VG/hadoop_LV /dir
Start the datanode services : start datanode

Check the hadoop cluster report :

So, we have successfully reduced the storage to 3GiB. Let’s check the Data inside file.txt created previously :

Wow!! The file and the Data inside it is safe!!
So, in this way we can provide elasticity to the Data Node storage.

You may also like

Leave a Comment