Tutorial: MQTT/Dashboard! Building an IOT Lightning Detector with your Raspberry Pi – Part 4
At SwitchDoc Labs we have been building a number of prototype IOT (Internet Of Things) devices for a number of different videos, articles, products and books we have been working on. Our latest Book, “Raspberry Pi IoT Projects: Prototyping Experiments for Makers
“, has been selling well on Amazon, published by APress-Springer/Daniel. Our first O’Reilly Publishing tutorial video, “Introduction to the ESP8266 and the IOT” also just came out. Time for a new project, the Raspberry Pi IOT Lightning Detector.
Part 4 shows you how to put together the IOT dashboard and the MQTT interface.
- - Part 1 - The Lightning IOT Project and How Does It Work?
- - Part 2 - Putting Together The Hardware
- - Part 3 - Understanding and Installing The Software
- - Part 4 - Installing the Internet Dashboard
- - Part 5 - Video Demonstration
The general idea of the Thunder Board IOT is to build a software platform to build more complex IOT sensors. In this column and the next, we will be going through a complete, IOT design. As well as providing a test bed for the new Grove Thunder Board Lightning Detector.
Adding an IOT Dashboard with PubNub and Freeboard
At SwitchDoc Labs, we have been using PubNub and Freeboard for several years to build our project MQTT software and Internet Dashboards. We are now building an interface for this product and others using MQTT and Alexa to provide a voice interface (for OurWeather see these posts on Instagram)
PubNub and Freeboard
We are now going to set up an IOT Dashboard using Pubnub and freeboard as shown in the block diagram when above.
PubNub is a free (at the base level) MQTT broker. It implements a publish/subscribe system for real time data streams. It is a very impressive service and tools and can grow with you as you develop either more complex applications or even products.
freeboard.io is a free (at the base level) dashboard application. You can build real-time, interactive dashboards and visualizations in minutes using the reasonably intuitive drag & drop interface. It’s not quite as pretty as the initialstate dashboard in Part 3, but it has an amazing amount of functionality under the hood and is open source so you can build your own blocks and modules for it.
Introduction to MQTT
MQTT is a publish-subscribe based “light weight” messaging protocol for use on top of the TCP/IP protocol, such as the WiFi packets that we are using in this project. It is designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited. The publish-subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message.
Publish–subscribe is a pattern where senders of messages, called publishers (in this case our ESP8266 is the publisher), don’t program the messages to be sent directly to subscribers, but instead characterize message payloads into classes without the specific knowledge of which subscribers the messages are sent to. Similarly, subscriberswill only receive messages that are of interest without specific knowledge of which publishers there are. Mosquitto operates as the broker in this system and routes the published data to the appropriate subscribers.
You can think of MQTT as writing stories for a newspaper where you don’t know who will be subscribing to the article.
JSON Data Payload
JSON is an open standard format that uses human-readable text to transmit data objects consisting of attribute–value pairs. It is the primary data format 
used for asynchronous browser/server communication, largely replacing XML. XML is a “heavier” protocol that is also hierarchical in nature, but with a great deal more redundancy that JSON. Yes, there are class wars going on for people that advocate JSON over XML, but in todays world of higher speed communication, it rarely matters. You can make the argument that the higher data density of JSON is a better choice for IOT applications.
Here is an example of the data packet we are using in the Raspberry Pi ThunderBoard IOT code in JSON for theLightning data payload:
{
"LastDistance": 6,
"LastPublishTimeStamp": "08:42:02 - 2018/01/02 PST",
"InterruptTimeStamp": "08:42:00 - 2018/01/02 PST",
"LastLightningResult": "Lightning! 6km away.",
"LastInterruptResult": 8,
"SoftwareVersion": "Pi003",
"LightningCount": 16,
"Noise_Floor": 1,
"Mask_Disturber": true,
"LightningTimeStamp": "08:42:00 - 2018/01/02 PST",
"Minimum_Strikes": 1,
"IndoorSet": false,
"InterruptCount": 19,
"LastResult": "Lightning! 6km away.",
"Display_LCO": false
}
The foundation of PubNub is providing you the ability to integrate scalable, realtime data streams into your applications. Using the publish/subscribe paradigm, subscribers to a particular channel will receive any and all messages that are published to that channel. It doesn’t matter if there is one subscriber, 10 subscribers, 1000 subscribers or millions of subscribers, a published message will be delivered to all of those subscribers on that channel in less than ¼ second1.
Setting up an Account at PubNub
Setup a free account at pubnub.com. Make sure you write down your password. Now let’s setup the MQTT info to connect up PiThunderBoardIOT.
Step 1) Create a new App by clicking on “NEW APP +”.. Type in ThunderBoardIOT in the “enter a new app” filed and hit the “CREATE NEW APP +” button. You should see this (except SunIOT will be ThunderBoardIOT):
Step 2) Click on the ThunderBoardIOT App page. The enter “ThunderBoardIOT” in the “Enter a New Keyset Name” field and hit the “CREATE NEW KEYSET +” button. That will get you a page that looks like this. Don’t worry about the application add-ons for this page. Those are all for more advanced (and cool) usage. Note the keys below will not work. They have all been disabled. You need to set up your own.
Next, let’s set up the ThunderBoardIOT Python Software for the IOT Dashboard.
Setting up the Pi ThunderBoard IOT Dashboard Software
If you followed Part 3 of this series, you already have the ThunderBoard IOT software running. Stop the software running at this point (especially if you have it running in the background) by doing this:
ps xaf | grep python sudo kill -9 (insert the process ID of the Thunderboard software)
Now we set up the publish and subscribe keys:
In a terminal window, do the following in your SDL_Pi_ThunderBoard_IOT directory.
cp config.py conflocal.py (this preserves your configuration changes when we update the software)
open conflocal.py in your favorite editor (we love “vi” but many use “nano”)
The file looks like this:
# # User configuration file # # to make your configuration file survive updates from Github, place your configuration into a file called conflocal.py # # # configuration for PubNub Pubnub_Publish_Key = "pub-XXXXX" Pubnub_Subscribe_Key = "sub-YYYYY"
Placing the Publish Key and Subscribe Key into the configuration strings above. It will then look something like this (use your own 
keys. THESE WILL NOT WORK!):
# configuration for PubNub Pubnub_Publish_Key = "pub-c-e77e14eb-2bb1-4f58-8ac7-cf4d75c46370" Pubnub_Subscribe_Key = "sub-c-10e3de76-a54c-11e6-8bfd-0619f8945a4f"
Now start theT hunderBoardIOT.py program running.
sudo python ThunderBoardIOT.py
If everything is hooked up right with PubHub, you should see something like this:
Publishing Data to PubNub time: 2018-01-02 09:06:48.105484 LastResult: None status.is_error False status.original_response [1, u'Sent', u'15149128082657315']
If you want to subscribe to your own data from PubHub, open another terminal wind on the Pi, configure listen.py with the same keys above and start listen.py
sudo python listen.py
Whenever ThunderBoard IOT publishes data, you are subscribing to the channel in listen.py. You can kill this process with “ctl-|” not with “ctl-c”.
pi@RPi3-65:~/SDL_Pi_ThunderBoard_IOT $ sudo python listen.py
incoming message {u'Noise_Floor': 0, u'Mask_Disturber': True, u'LastDistance': 14, u'LastPublishTimeStamp': u'09:11:49 - 2018/01/02 PST', u'InterruptTimeStamp': u'09:11:47 - 2018/01/02 PST', u'LastLightningResult': u'Lightning! 14km away.', u'LightningTimeStamp': u'09:11:47 - 2018/01/02 PST', u'LastInterruptResult': 8, u'Minimum_Strikes': 1, u'SoftwareVersion': u'Pi003', u'IndoorSet': False, u'InterruptCount': 14, u'LastResult': u'Lightning! 14km away.', u'LightningCount': 12, u'Display_LCO': False}
Setting up an Account at Freeboard
Now we have all the data that we want being published to the PubNub MQTT broker, now it is time to set up an account at freeboard.io.
Step 1) Set up your free account at freeboard.io
Step 2) Go to My Freeboards and create a new Freeboard called SDL_PI_ThunderBoard_IOT. Click on the name of SDL_PI_ThunderBoard_IOT to begin building your IOT Dashboard. Yes, we know we have a lot of other projects on Freeboard. You won’t to begin with.
The JSON for this dashboard is below. It is included in the GitHub repository. Being able to import dashboards is critical to being able to distribute kits and repositories for projects like this. You can either build your own dashboard following tutorial or copy and paste the following JSON into a file called “dashboard.json” and importing it from the Freeboard site. You will have to change your subscribe key. This one won’t work.
{"version":1,"allow_edit":true,"plugins":["/plugins/all"],"panes":[{"title":"Last Lightning Event","width":1,"row":{"3":1},"col":{"3":2},"col_width":"2","widgets":[{"type":"text_widget","settings":{"title":"Last Lightning Stroke","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"LightningTimeStamp\"]","animate":true}},{"type":"text_widget","settings":{"title":"Last Stroke Distance","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"LastDistance\"]","animate":true,"units":"km"}},{"type":"text_widget","settings":{"title":"Lightning Strokes Counted","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"LightningCount\"]","animate":true}}]},{"title":"The Pi THUNDERBOARD IOT","width":1,"row":{"3":1,"4":11},"col":{"3":1,"4":1},"col_width":1,"widgets":[{"type":"plugin52d968dc3239abe64b000006","settings":{"src":"https://www.switchdoc.com/wp-content/uploads/2017/11/lightning-bolt-picture-1.jpg","refresh":0}}]},{"width":1,"row":{"3":9,"4":9},"col":{"3":2,"4":2},"col_width":1,"widgets":[{"type":"gauge_widget","settings":{"title":"Lightning Distance","value":"datasources[\"ThunderBoard IOT\"][\"LastDistance\"]","units":"Km","min_value":"0","max_value":"30"}},{"type":"text_widget","settings":{"title":"Software Version","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"SoftwareVersion\"]","animate":true}}]},{"width":1,"row":{"3":9,"4":9},"col":{"3":3,"4":3},"col_width":1,"widgets":[{"type":"plugin52d968dc3239abe64b000006","settings":{"src":"https://www.switchdoc.com/wp-content/uploads/2017/11/LightningBox.png","refresh":0}}]},{"title":"Thunder Board Configuration","width":1,"row":{"3":11,"4":1},"col":{"3":1,"4":1},"col_width":1,"widgets":[{"type":"indicator_widget","settings":{"title":"Indoor Set","value":"datasources[\"ThunderBoard IOT\"][\"IndoorSet\"]","on_text":"1","off_text":"0"}},{"type":"gauge_widget","settings":{"title":"Noise Floor","value":"datasources[\"ThunderBoard IOT\"][\"Noise_Floor\"]","min_value":0,"max_value":"7"}},{"type":"text_widget","settings":{"title":"Minimum # Strikes Before Report","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"Minimum_Strikes\"]","animate":true}},{"type":"indicator_widget","settings":{"title":"Distrubers Masked","value":"datasources[\"ThunderBoard IOT\"][\"Mask_Disturber\"]","on_text":"1","off_text":"0"}}]},{"width":1,"row":{"3":19,"4":19},"col":{"3":2,"4":2},"col_width":"2","widgets":[{"type":"text_widget","settings":{"title":"Last Message Received","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"LastPublishTimeStamp\"]","animate":true}}]},{"width":1,"row":{"3":23,"4":19,"5":19},"col":{"3":2,"4":2,"5":2},"col_width":"2","widgets":[{"type":"plugin52d968dc3239abe64b000006","settings":{"src":"https://www.switchdoc.com/wp-content/uploads/2015/04/Untitled-design-4.png","refresh":0}}]},{"title":"Last Results from Thunder Board","width":1,"row":{"3":25,"4":9,"5":9,"6":9},"col":{"3":1,"4":3,"5":3,"6":3},"col_width":1,"widgets":[{"type":"text_widget","settings":{"title":"Last Result","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"LastResult\"]","animate":true}},{"type":"text_widget","settings":{"title":"Last Result TimeStamp","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"InterruptTimeStamp\"]","animate":true}}]},{"width":1,"row":{"3":31,"6":31},"col":{"3":1,"6":1},"col_width":1,"widgets":[{"type":"text_widget","settings":{"title":"Software Version","size":"regular","value":"datasources[\"ThunderBoard IOT\"][\"SoftwareVersion\"]","animate":true}}]}],"datasources":[{"name":"ThunderBoard IOT","type":"plugin5376758af1776c1c2e000326","settings":{"subscribe_key":"sub-c-c735f332-ccc7-11e7-a7e0-ba9127ff21d1","channel":"ThunderBoardIOT"}}],"columns":3,"pane_header_bg_color":null,"pane_bg_color":null}
Step 3) On your new board, click import and load the dashboard.json file that is in your SDL_ThunderBoard_IOT directory.
You will then see this. Click on the “wrench” on top of the screen.
Step 3) Edit the Datasource. Click ThunderBoard IOT under DATASOURCES In this case, the Datasource is our PubNub MQTT broker. This is very important to get right. Fill out the form as below, using your subscribe key from PubNub. The key in the JSON file WILL NOT WORK.
You are done. If you have the ThunderBoard Python running on your Pi, it will eventually repopulate. You can always restart the ThunderBoardIOT python software on your Pi to force a repopulation immediately. If it doesn’t work, check your keys very carefully. Look at your listen.py output.
You should be seeing something like this:
You can change your map and graphics to suit your own tastes.
Conclusion
Now you have built your very own Lighting Detector IOT device. You can share the dashboard with other people and modify it to your hearts content. Please modify the software and boards. You learn a lot more by doing that.
Stay tuned for the Arduino version of the IOT device and also new Alexa interfaces for this and other devices.