DevOps(7)Spark Deployment on VM with Kafka

1397535668

DevOps(7)Spark Deployment on VM with Kafka
 
1. Install the zookeeper
I pick up this version http://apache.mirrors.lucidnetworks.net/zookeeper/stable/zookeeper-3.4.6.tar.gz.
 
> sudo ln -s /home/carl/tool/zookeeper-3.4.6 /opt/zookeeper-3.4.6
> sudo ln -s /opt/zookeeper-3.4.6 /opt/zookeeper
 
Add it to the path
> cp conf/zoo_sample.cfg conf/zoo.cfg
 
Start the server
> zkServer.sh start zoo.cfg
 
Connect to the server
> zkCli.sh -server localhost:2181
 
Stop the server
> zkServer.sh stop
 
2. Install kafka
I pick up this version http://www.carfab.com/apachesoftware/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz
 
Build the source
> ./gradlew -PscalaVersion=2.10.0 releaseTarGz -x signArchives
 
Find the binary file here
> cd core/build/distributions/
 
Place the binary file to working directory and add it to the path.
 
Adjust the config file  vi config/server.properties 
 
Command to start kafka
nohup bin/kafka-server-start.sh config/server-dev2.properties &
 
Install the kafka monitor
> git clone https://github.com/quantifind/KafkaOffsetMonitor.git
 
Build and generate the assembly jar 
> sbt clean update
> sbt assembly
 
Similar command to start that
> java -cp /opt/kafka-monitor/lib/*.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk ubuntu-dev1,ubuntu-dev2 --port8082--refresh 10.seconds --retain 2.days
 
Tip:
Some Mysql Timezone Config
> mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -u root -p mysql
 
Or
 
mysql_tzinfo_to_sql /usr/share/zoneinfo/ > mysql.sql
 
Fix the too long string manually
> cat mysql.sql | mysql -u root -p mysql
 
3. Install rabbitMQ
http://sillycat.iteye.com/admin/blogs/2183555
http://sillycat.iteye.com/blog/1565771
 
Install erlang from source
>sudo apt-get install build-essential libncurses5-dev openssl libssl-dev fop xsltproc unixodbc-dev
 
http://www.erlang.org/download/otp_src_17.4.tar.gz
 
Install rabbitmq from source
http://www.rabbitmq.com/releases/rabbitmq-server/v3.4.3/rabbitmq-server-generic-unix-3.4.3.tar.gz
 
Command to start rabbitmq
>sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 sbin/rabbitmq-server -detached
 
When I use guest/guest to login, I get this error message:
=ERROR REPORT==== 5-Feb-2015::13:14:08 ===
webmachine error: path="/api/whoami"
"Unauthorized"
 
Solution:
That is a new feature, I need to solve it like this
http://stackoverflow.com/questions/22850546/cant-access-rabbitmq-web-management-interface-after-fresh-install
 
Creating User, set group, set permission
> rabbitmqctl -n rabbit1 add_user sillycat xxxxx
> rabbitmqctl -n rabbit1 set_user_tags sillycat administrator
> rabbitmqctl -n rabbit1 set_permissions -p / sillycat ".*" ".*" ".*"
 
Start the App and Visit the console
cd /opt/rabbitmq
sudo RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit2 sbin/rabbitmq-server -detached

enable cluster

sudo sbin/rabbitmqctl -n rabbit2 stop_app
sudo sbin/rabbitmqctl -n rabbit2 join_cluster rabbit1@ubuntu-dev1
sudo sbin/rabbitmqctl -n rabbit2 start_app

visit page
http://ubuntu-dev2:15672
 
References:
http://sillycat.iteye.com/blog/2166583
http://sillycat.iteye.com/blog/2167216
 
http://sillycat.iteye.com/blog/2015175
 
http://sillycat.iteye.com/blog/2066116