Setting up Storm Cluster in Local Machine

TechySoul Java 0 Comments

You have found the right place if you are looking for a step-by-step complete guide to setup Apache storm from scratch on your local machine and deploy a topology jar onto the same.

Step 1 : Setting up Installation Directory

Create a new directory ‘DevApps’ to install all required applications within the directory of your choice. This new directory would be referred as $INSTALL_DIR hereafter.

Tip for Windows Users
There should not be any space in the full path of this directory.
If Windows is installed on C drive then this directory should look like – C:/DevApps.

Step 2 : Installing required applications

1. Java 8 SE Development Kit – 1.8.0_121
Download from here and install in $INSTALL_DIR
2. Python 2.7.13
Download from here and select $INSTALL_DIR for installation.
3. Apache Zookeeper – 3.4.9
Download from here and just extract to $INSTALL_DIR to install.
4. Apache Storm – 1.0.3
Download from here and extract to $INSTALL_DIR

Step 3 : Setting up Environment Variables

[Windows]
Add/Update following variables as below. Please note the version of Java SDK and Apache Storm that you have used for installation and make changes accordingly if needed.

JAVA_HOME = C:/DevApps/Java/jdk1.8.0_121
STORM_HOME = C:/DevApps/apache-storm-1.0.3

Add following to the existing PATH variable

PATH = %STORM_HOME%/bin;%JAVA_HOME/%bin;C:/DevApps/Python27;C:/DevApps/Python27/Lib/site-packages;C:/DevApps/Python27/Scripts;

[MacOS or Unix]

export STORM_HOME = ~/DevApps/apache-storm-1.0.3

Step 4 : Configuring local storm cluster

Create another folder storm-local in DevApps to be used as Storm local directory and use it in the next step.
Make following changes to storm.yaml file from %STORM_HOME%/conf folder
[Windows]

storm.zookeeper.servers:
     - "localhost"
nimbus.host: "localhost"
storm.local.dir: "C:/DevApps/storm-local"
[MacOS or Unix]
storm.zookeeper.servers:
     - "localhost"
nimbus.host: "localhost"
storm.local.dir: "~/DevApps/storm-local"

Create a working config for ZooKeeper using the priovided sample.
Goto $INSTALL_DIR/zookeeper-3.4.9/conf
and Copy/Rename zoo_sample.cfg as zoo.cfg

Step 5 : Starting local storm cluster

Let’s first start Zookeeper
[Windows ]

zkServer.cmd

[MacOS or Linux]

cd ~/DevApps/zookeeper-3.4.9/bin
./zkServer.sh start

Start Nimbus, Supervisor and Storm UI
[Windows]

storm nimbus
storm supervisor
storm ui

[MacOS or Linux]

cd ~/DevApps/apache-storm-1.0.3/bin
./storm nimbus &
./storm supervisor &
./storm ui &

Verify the setup by browsing to following link using any browser of your choice.

http://blog.techysoul.com:8080

Step 6 : Deploying Topology

The syntax of the command deploys the topology in the storm cluster is as below.

storm jar <absolute_path_of_jar> <fully_qualified_name_of_main_class> <name_of_topology>

Let’s take an example to understand this –

absolute_path_of_jar = ~workspace/storm-trade-processing/target/storm-trade-processing-1.0-SNAPSHOT.jar

fully_qualified_class_name = com.techysoul.apache.storm.tradeprocessing.app.TradeProcessingTopology

name_of_topology = TradeProcessingTopology

Following command deploys the topology in the storm cluster.

cd ~/DevApps/apache-storm-1.0.3/bin
./storm jar ~/workspace/storm-trade-processing/target/storm-trade-processing-1.0-SNAPSHOT.jar
com.techysoul.apache.storm.tradeprocessing.app.TradeProcessingTopology TradeProcessingTopology

Step 7 : Graceful Exit

cd ~/DevApps/zookeeper/bin
./zkServer.sh status
./zkServer.sh stop
./zkServer.sh status

Please let me know if you get stuck at any step while following this tutorial. Happy Storming!

Share this Post

Leave a Reply

Your email address will not be published. Required fields are marked *