building a c# storm topology (yes, it is a jvm-based framework)

Blog post
added by
Lester Martin

dkfjdfkjkdfj

kdjfkdjfkfj

JUST CAPTURING NOTES OF STEPS I WENT THROUGH TO GET APACHE STORM TOPOLOGY BUILD WITH C# AND THEN RUNNING ON AZURE HD-INSIGHTS.

Used instructions documented at https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-tutorial-get-started-linux to spin up an HDInsight cluster.  Thankfully, the estimate came in a bit cheaper than on that web page, but it is still expensive. 

Regarding this cost, and as clearly documented in the setup instructions, the problem with the HDInsight cluster for testing like this is that you can’t stop the meter.  Yes, you can stop the machines that the cluster is running on, but that doesn’t stop the charge clock, so you have to tear it all down when you want to stop being charged.  Fortunately, it is easy to spin up a new one.  Creating my second cluster took 10 mouse clicks, filling out two textboxes (cluster name and ssh user’s password), and less than four minutes of time.  The actual provisioning of the cluster took a bit less than 20 minutes.

For those familiar with Hortonworks, the Ambari and Storm UI’s that come up make you feel right at home!!

 

The HDInsights setup instructions show you how to deploy a pre-built WordCount topology and to stop it once you have finished exploring a bit as highlighted below.

HW13005:~ lmartin$ ssh sshuser@myhdinsight-ssh.azurehdinsight.net
sshuser@hn0-myhdin:~$ storm jar /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar org.apache.storm.starter.WordCountTopology wordcount
32492 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: wordcount
sshuser@hn0-myhdin:~$ storm list
Topology_name        Status     Num_tasks  Num_workers  Uptime_secs
-------------------------------------------------------------------
wordcount            ACTIVE     29         3            634       
sshuser@hn0-myhdin:~$ storm kill wordcount
6116 [main] INFO  o.a.s.c.kill-topology - Killed topology: wordcount
sshuser@hn0-myhdin:~$ 

kdfkdfjkjdf

For a dev environment, I used https://buildazure.com/2017/03/07/visual-studio-2017-development-using-a-vm-in-azure/ as my setup guide.  I was reminded to fully stop (from the Azure portal) the VM as described in https://buildazure.com/2017/03/16/properly-shutdown-azure-vm-to-save-money/.  Fortunately, after a month or so of just messing around my bill is < $20.

First up was using instructions at https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-deploy-monitor-topology#hdinsight-tools-for-visual-studio to build & deploy a sample topology.

THIS NEXT LITTLE BIT IS FOR ME AND CAN BE DELETED…

At this point, here are the Azure “resources” I have; several of these can be reused when I start up a new Storm cluster (I think).

The “VS-Storm” stuff is for my dev box, but the other stuff should be for my HDI-based Storm cluster.

Now it is time to develop/build/deploy a topology and https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-develop-csharp-visual-studio-topology was a big help.

NOTE: See https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-develop-java-topology for help building a Java topology as well as my YouTube video on the topic.

dkjfkdjf

Leave a Comment

Your email address will not be published. Required fields are marked *