Hadoop and Spark Installation on Raspberry Pi-3 Cluster – Part-4

In this part we will see the configuration of Slave Node. Here are the steps

1. Mount second Raspberry Pi-3 device on the nylon standoffs (on top of Master Node)
2. Load the image from part2 into a sd_card
3. Insert the sd_card into one Raspberry Pi-3 (RPI) device
4. Connect RPI to the keyboard via USB port
5. Connect to monitor via HDMI cable
6. Connect to Ethernet switch via ethernet port
7. Connect to USB switch via micro usb slot
8. Hadoop related changes on Slave node

Here Steps1-7 are all physical and hence I am skipping them.

Once the device is powered on, login via external keyboard and monitor and change the hostname from rpi3-0 (which comes from base image) to rpi3-1

Step #8: Hadoop Related Configuration

• Setup HDFS

 
sudo mkdir -p /hdfs/tmp  
sudo chown hduser:hadoop /hdfs/tmp  
chmod 750 /hdfs/tmp  
hdfs namenode -format

• Update /etc/hosts file

 
127.0.0.1	localhost
192.168.2.1	rpi3-0
192.168.2.101	rpi3-1
192.168.2.102	rpi3-2
192.168.2.103	rpi3-3

• Repeat the above steps for each of the slave node. And for every addition of slave node, ensure
• ssh is setup from master node to slave node
• slaves file on master is updated
• /etc/hosts file on both master and slave is updated

Start the hadoop/spark cluster

• Start dfs and yarn services

 
cd /opt/hadoop-2.7.3/sbin 
start-dfs.sh 
start-yarn.sh 

• On master node “jps” should show following

 
hduser@rpi3-0:~ $ jps
20421 ResourceManager
20526 NodeManager
19947 NameNode
20219 SecondaryNameNode
24555 Jps
20050 DataNode

• On Slave Node “jps” should show following processes

 
hduser@rpi3-3:/opt/hadoop-2.7.3/logs $ jps
2294 NodeManager
2159 DataNode
2411 Jps

• To verify the successful installation, run a hadoop and spark job in cluster mode and you will see the Application Master tracking URL.
• Run spark Job
  • spark-submit –class com.learning.spark.SparkWordCount –master yarn –executor-memory 512m ~/word_count-0.0.1-SNAPSHOT.jar /ntallapa/word_count/text 2
• Run example mapreduce job
  • hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /ntallapa/word_count/text /ntallapa/word_count/output

Hadoop and Spark Installation on Raspberry Pi-3 Cluster – Part-3

In this part we will see the configuration of Master Node. Here are the steps

1. Mount first Raspberry Pi-3 device on the nylon standoffs
2. Load the image from part2 into a sd_card
3. Insert the sd_card into one Raspberry Pi-3 (RPI) device
4. Connect RPI to the keyboard via USB port
5. Connect to monitor via HDMI cable
6. Connect to Ethernet switch via ethernet port
7. Connect to USB switch via micro usb slot
8. DHCPD Configuration
9. NAT Configuration
10. DHCPD Verification
11. Hadoop related changes on Master node

Here Steps1-7 are all physical and hence I am skipping them.

Step #8: dhcpd configuration

This node will serve as DHCP server or NAT server and overall controller of the cluster

• Goto “sudo raspi-config” -> Advanced Options -> HostName -> “rpi3-0” (make sure its rpi3-0 as its our first node)
• sudo apt-get install isc-dhcp-server
• sudo nano /etc/dhcp/dhcpd.conf
• Define subnet which will be the network that all the RPI-3 nodes connect to.

 
subnet 192.168.2.0 netmask 255.255.255.0 {
        range 192.168.2.100 192.168.2.200;
        option broadcast-address 192.168.2.255;
        option routers 192.168.2.1;
        max-lease-time 7200;
        option domain-name "rpi3";
        option domain-name-servers 8.8.8.8;
}

• Adjust server configuration
• sudo nano /etc/default/isc-dhcp-server
• Tell which interface to use at last line. (“eth0”)

 
# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
#       Separate multiple interfaces with spaces, e.g. "eth0 eth1".
INTERFACES="eth0"

• Configure the interfaces file of rpi3-0 in order to be served as dhcp server and nat server for the rest of the pi cluster
• sudo nano /etc/network/interfaces
• Make the below changes and reboot the PI

 
auto eth0
iface eth0 inet static
	address 192.168.2.1
	netmask 255.255.255.0

Step #9: NAT configuration

• Now we will configure IP tables to provide Network Address Translation services on our master node rpi3-0
• sudo nano /etc/sysctl.conf
• uncomment “net.ipv4.ip_forward=1”
• sudo sh -c “echo 1 > /proc/sys/net/ipv4/ip_forward”
• Now it has been activated, run below 3 commands to configure IP Tables correctly

 
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED
sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT

• Make sure we have this setup correct
• sudo iptables -t nat -S
• sudo iptables -S
• In order to avoid loosing this config upon reboot, do
  • sudo sh -c “iptables-save > /etc/iptables.ipv4.nat” (save iptables configuration to a file)
  • sudo nano /etc/network/interfaces (add below line to interfaces file)
    post-up iptables-restore < /etc/iptables.ipv4.nat

 
auto eth0
iface eth0 inet static
	address 192.168.2.1
	netmask 255.255.255.0
	post-up iptables-restore  < /etc/iptables.ipv4.nat

Step #10: Verify dhcpd

• To see the address that has been assigned to the new PI
• cat /var/lib/dhcp/dhcpd.leases
• This would also give us the MAC address of the newly added node
• It is always handy to have the dhcp server assign fixed addresses to each node in the cluster so that its easy to remember the node by ipaddress. For instance next node in the cluster is rpi3-1 and it would be helpful to have a ip 192.168.2.101. To do this modify dhcp server config file
  • sudo nano /etc/dhcp/dhcpd.conf

   
  host rpi3-1 {
      hardware ethernet MAC_ADDRESS;
      fixed-address 192.168.2.101;
  }
  

  • Eventually this file will have a entry to all the nodes in the cluster
  • Now we can ssh into the new node via IP Address

Step #11: Hadoop Related Configuration

• Setup SSH

 
su hduser 
cd ~  
mkdir .ssh  
ssh-keygen  
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  
chmod 0750 ~/.ssh/authorized_keys  
// Lets say we added new slave node rpi3-1, then copy the ssh id to the slave node which will enable passwordless login
ssh-copy-id hduser@rpi3-1 (Repeat for each slave node)  
ssh hduser@rpi3-1

• Setup HDFS

 
sudo mkdir -p /hdfs/tmp  
sudo chown hduser:hadoop /hdfs/tmp  
chmod 750 /hdfs/tmp  
hdfs namenode -format

• Edit master and slave config files
  • sudo nano /opt/hadoop-2.7.3/etc/hadoop/masters

   
  rpi3-0
  

  • sudo nano /opt/hadoop-2.7.3/etc/hadoop/slaves

   
  rpi3-0
  rpi3-1
  rpi3-2
  rpi3-3
  

• Update /etc/hosts file

 
127.0.0.1	localhost
192.168.2.1	rpi3-0
192.168.2.101	rpi3-1
192.168.2.102	rpi3-2
192.168.2.103	rpi3-3

References:
https://learn.adafruit.com/setting-up-a-raspberry-pi-as-a-wifi-access-point/install-software

Hadoop and Spark Installation on Raspberry Pi-3 Cluster – Part-2

Cluster Architecture

• Master Node will be connected to home router via WiFi
• Master Node to Slave Node connection will be established through Ethernet switch via Ethernet Cables
• From my MAC (which will be on my home network), I will be able to SSH to the master node and then control the whole cluster

For Spark/Hadoop Cluster, there are few more TODOs that we need to take care of

• Update /etc/hosts on every node (master and slave) with hostname and ip_address of every other node
• Use same super user and group to do all installations on every node
• Enable SSH on every node and establish passwordless SSH communication from Master to every Slave node.
• Install zip/unzip and java on every node.

In this part we will see SINGLE NODE SETUP. I will be using MAC to perform all these steps.

Step #1: Load Raspbian Pi image onto the MicroSD Card

• Download SD Formatter from https://www.sdcard.org/downloads/formatter_4/eula_mac/index.html
• Format the disk (follow steps from google)
• Download the Raspbian_Lite OS from https://www.raspberrypi.org/ and follow these instructions
  • diskutil list (this will list the newly added disk in my case it was /dev/disk4)
  • diskutil unmountDisk /dev/disk4
  • sudo dd bs=1m if=~/Downloads/2017-01-11-raspbian-jessie-lite.img of=/dev/rdisk4

Step #2: Configure the PI, connect to WiFi and upgrade all latest patches.

• sudo raspi-config
  • Change pwd
  • Localization options (change locale to us_en, timezone to US-Eastern and wifi country to US)
  • Advanced Options – Mem Split from 64 to 16(becos of rasbian_lite OS will take very less footprint and it does not have any UI)
  • Interfacing Options – Enable SSH
• sudo vi /etc/network/interfaces
  • Ethernet eth0 is the wired connection which we will be using
  • wlan0 is the wifi adapter on the board: Configure wifi so that we can use wifi to get updates on the Raspbian Lite OS
  • change manual to dhcp => this tells the interface that we get the settings via dhcp AND add SSID and PWD of home router
    • change the line “iface wlan0 inet manual” to

 
iface wlan0 inet dhcp
    wpa-ssid "SSID/NETWORK_NAME"
    wpa-psk "PASSWORD"
 

•  Reboot the interface
  • sudo ifdown wlan0
  • sudo ifup wlan0
• Now Raspberry PI is connected to WIFI
• Update and Upgrade the Raspberry PI
  • sudo apt-get update
  • sudo apt-get upgrade

Step #3: Create separate superuser and group

• We will use this user and group for all our core installations and configuration changes on all nodes

 
sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
sudo adduser hduser sudo
su hduser
 

Step #4: Download and Install REQUIRED SOFTWARES

• Download and install zip utility
  • sudo apt-get install zip unzip
• Download and install java
  • sudo apt-get install oracle-java7-jdk
• Download, install and configure Spark

wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz
    sudo tar -xvzf spark-2.1.0-bin-hadoop2.7.tgz -C /opt/
    sudo chown -R hduser /opt/spark-2.1.0-bin-hadoop2.7
    source ~/.bashrc
    cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh  
        SPARK_MASTER_IP=192.168.2.1
        SPARK_WORKER_MEMORY=512m

• Download, install and configure Hadoop

  wget http://apache.osuosl.org/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
  sudo mkdir /opt
  cd ~
  sudo tar -xvzf hadoop-2.7.1.tar.gz -C /opt/
  cd /opt
  sudo chown -R hduser:hadoop hadoop-2.7.1/
  sudo nano /opt/hadoop-2.7.3/etc/hadoop/
  export JAVA_HOME=/usr/lib/jvm/jdk-7-oracle-arm-vfp-hflt/jre
  

• sudo nano /opt/hadoop-2.7.3/etc/hadoop/hdfs-site.xml

   
          <property>
                  <name>dfs.replication</name>
                  <value>1</value>
          </property>
  

• sudo nano /opt/hadoop-2.7.3/etc/hadoop/core-site.xml

   
          <property>
                  <name>fs.default.name</name>
                  <value>hdfs://rpi3-0:54310</value>
          </property>
          <property>
                  <name>hadoop.tmp.dir</name>
                  <value>/hdfs/tmp</value>
          </property>
  

• sudo nano /opt/hadoop-2.7.3/etc/hadoop/mapred-site.xml

   
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
    <property>
      <name>mapreduce.map.memory.mb</name>
      <value>256</value>
    </property>
    <property>
      <name>mapreduce.map.java.opts</name>
      <value>-Xmx204m</value>
    </property>
    <property>
      <name>mapreduce.reduce.memory.mb</name>
      <value>102</value>
    </property>
    <property>
      <name>mapreduce.reduce.java.opts</name>
      <value>-Xmx102m</value>
    </property>
    <property>
      <name>yarn.app.mapreduce.am.resource.mb</name>
      <value>128</value>
    </property>
    <property>
      <name>yarn.app.mapreduce.am.command-opts</name>
      <value>-Xmx102m</value>
    </property>
  

• sudo nano /opt/hadoop-2.7.3/etc/hadoop/yarn-site.xml

   
          <property>
                  <name>yarn.resourcemanager.resource-tracker.address</name>
                  <value>rpi3-0:8025</value>
          </property>
          <property>
                  <name>yarn.resourcemanager.scheduler.address</name>
                  <value>rpi3-0:8030</value>
          </property>
          <property>
                  <name>yarn.resourcemanager.address</name>
                  <value>rpi3-0:8050</value>
          </property>
          <property>
                  <name>yarn.nodemanager.aux-services</name>
                  <value>mapreduce_shuffle</value>
          </property>
          <property>
                  <name>yarn.nodemanager.resource.cpu-vcores</name>
                  <value>4</value>
          </property>
          <property>
                  <name>yarn.nodemanager.resource.memory-mb</name>
                  <value>1024</value>
          </property>
          <property>
                  <name>yarn.scheduler.minimum-allocation-mb</name>
                  <value>128</value>
          </property>
          <property>
                  <name>yarn.scheduler.maximum-allocation-mb</name>
                  <value>1024</value>
          </property>
          <property>
                  <name>yarn.scheduler.minimum-allocation-vcores</name>
                  <value>1</value>
          </property>
          <property>
                  <name>yarn.scheduler.maximum-allocation-vcores</name>
                  <value>4</value>
          </property>
          <property>
                  <name>yarn.nodemanager.vmem-check-enabled</name>
                  <value>false</value>
          </property>
          <property>
                  <name>yarn.nodemanager.pmem-check-enabled</name>
                  <value>true</value>
          </property>
          <property>
                  <name>yarn.nodemanager.vmem-pmem-ratio</name>
                  <value>4</value>
          </property>
  

• Add environment variables to the bashrc file
  sudo nano ~/.bashrc

  export JAVA_HOME=/usr/lib/jvm/jdk-7-oracle-arm-vfp-hflt/jre
  export HADOOP_HOME=/opt/hadoop-2.7.3
  export HADOOP_MAPRED_HOME=$HADOOP_HOME
  export HADOOP_COMMON_HOME=$HADOOP_HOME
  export HADOOP_HDFS_HOME=$HADOOP_HOME
  export YARN_HOME=$HADOOP_HOME
  export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
  
  export SPARK_HOME=/opt/spark-2.1.0-bin-hadoop2.7 
  export PATH=$PATH:$SPARK_HOME/bin  
  

Step #5: Create image from SD Card and clone it to all other PIs on the cluster

• Switch off the Raspberry Pi-3 and take the sd-card out and plug it into mac
• Run the below commands
  • diskutil list
  • sudo dd if=/dev/disk4 of=~/Downloads/raspberrypi_base_with_hdp.dmg
• On other nodes
  • diskutil unmountDisk /dev/disk4
  • sudo dd bs=1m if=~/Downloads/raspberrypi_base_with_hdp.dmg of=/dev/rdisk4

References:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

A Raspberry Pi Hadoop Cluster with Apache Spark on YARN: Big Data 101

Hadoop and Spark Installation on Raspberry Pi-3 Cluster – Part-1

In this part we will go through the hardware configurations of the Raspberyy PI-3 4 Node Cluster setup.

Hardware Configuration

1. Raspberry Pi 3 Model B 2016 Single Board Computer – 4 = 120 (bought from Microcentre)

2. Samsung EVO 32GB Class 10 Micro SDHC Card with Adapter (MB-MP32DA/AM) – 4 (12) = $48

3. Cable Matters 8-Pack, Cat5E Snagless Ethernet Patch Cable in Blue 3 Feet – 4 = $10

4. TRENDnet 8-Port Unmanaged 10/100 Mbps GREENnet Ethernet Desktop Plastic Housing Switch,TE100-S8 – 1 = $15

5. Anker [4-Pack] PowerLine Micro USB (1ft) – 1 = $10

6. StarTech.com USB2TYPEM 3 Feet USB to Type M Barrel 5V DC Power Cable – 1 = $4

7. 100 Pcs M2.5 x 10mm + 6mm PC Board Hexagonal Hex Threaded Spacer – 1 = $14

8. ORICO DUB-10P-WH 96W 10 Ports Family-Sized Smart Super USB Charger with 10 × 5V 2.4A Port – White – 1 = $20

9. Machine Screws (bought from Home Depot) – 2 = under $2

10. Thin card board from Home Depot – $4

Total Budget under $250

Except Machine Screws, Card Board and RaspberryPis all above things are bought in Amazon and NewEgg.

Additionally you can buy heat sinks and fans based on the load and extent you are going to use this cluster.

My personal cluster setup Steps:

• Drill 4 holes on to the card board as per the holes available on the Raspberry Pi
• Stack two nylon standoffs together form 20 such sets
• Insert screws from below the cardboard and 4 stacked nylon standoffs from above the cardboard
• Keep stacking Raspberry Pis (RPIs) one by one and tighten the standoffs.
• Connect 4 micro USB cables from RPIs to the USB switch
• Connect 4 ethernet cables from RPIs to the Ethernet switch
• Connect Ethernet switch to the USB switch via USB2TYPEM cable.
• Insert 32G samsung sd card into each of the RPIs
• Switch on the USB switch and the entire cluster will be powered.