Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board – DEVELOPPARADISE
23/05/2018

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board


Intro

This article explores the method for setting up a time series for sensor data using the UP Squared* board and Grove shield, the Amazon Web Services (AWS)* Greengrass, and Plotly*. First, we will collect the data from Grove’s UV sensor then we will create a time series using Plotly. Finally, we will publish the time series URL to Greengrass’s IoT Cloud using the AWS Lambda function.

Learn more about the AWS* Greengrass

Learn more about the UP Squared board

Learn more about Plotly

Prerequisites

UP Squared board:

AWS Greengrass:

Plotly:

AWS Greengrass

To install AWS Greengrass, follow these instructions

Check that you have installed all the needed dependencies:

sudo apt update git clone https://github.com/aws-samples/aws-greengrass-samples.git cd aws-greengrass-samples cd greengrass-dependency-checker-GGCv1.3.0 sudo ./check_ggc_dependencies
Code 1. Commands to Check AWS Dependencies

On UP Squared board, start Greengrass:

cd <path-to-greengrass-folder>/ggc/core sudo ./greengrassd start
Code 2. Commands to Start AWS Greengrass

Time Series

On UP Squared board, install Plotly and its dependencies:

sudo pip install pandas sudo pip install flask sudo apt-get install sqlite3 qlite3-dev sudo pip install plotly
Code 3. Commands to Plotly with Dependencies

To update Plotly:

sudo pip install plotly --upgrade
Code 4. Commands to Update Plotly

Go to your home directory and create a new directory:

cd ~ mkdir .plotly
Code 5. Commands to Create a Directory

Get the API key and your login info from the Plotly account page. Create .credentials file in the .plotly directory:

{     "username": "your-username",     "stream_ids": [],     "api_key": "your-api-key" }
Code 6. .credentials file

You will need your username and API key information in the next section to authenticate with Plotly.

Grove Sensors

To interface with Grove sensors, install MRAA and UPM libraries:

sudo add-apt-repository ppa:mraa/mraa sudo apt-get update sudo apt-get install libmraa1 libmraa-dev mraa-tools python-mraa python3-mraa sudo apt-get install libupm-dev libupm-java python-upm python3-upm node-upm upm-example
Code 7. Commands to Install Grove Dependencies

UV Sensor

This section shows how to collect data with the Grove UV sensor and save the sensor data in a CSV. The second script reads the CSV file and creates a time series using Plotly and Pandas Python* packages. The URL for time series is saved in a file that is used by the AWS Lambda function in the next section. Code used in this section was modified from the source code, which can be found here. Here’s the modified code we will use to retrieve and save the sensor data:

from __future__ import print_function import time, sys, signal, atexit ​from datetime import datetime from upm import pyupm_si114x as upmSi114x import mraa  def main():     # Interface with sensors through the Grove shield     mraa.addSubplatform(mraa.GROVEPI, "0")      # Instantiate a SI114x UV Sensor on I2C bus 0     myUVSensor = upmSi114x.SI114X(0)      ## Exit handlers ##     # This stops python from printing a stacktrace when you hit control-C     def SIGINTHandler(signum, frame):         raise SystemExit      # This function lets you run code on exit,     # including functions from myUVSensor     def exitHandler():         print("Exiting")         sys.exit(0)      # Register exit handlers     atexit.register(exitHandler)     signal.signal(signal.SIGINT, SIGINTHandler)      # First initialize it     myUVSensor.initialize()      print("UV Index Scale:")     print("---------------")     print("11+        Extreme")     print("8-10       Very High")     print("6-7        High")     print("3-5        Moderate")     print("0-2        Low/n")      # update every second and print the currently measured UV Index     while (1):         # update current value(s)         myUVSensor.update()         uv = myUVSensor.getUVIndex()         # print detected value         print("UV Index:", uv)         dt = datetime.now()          with open("uv_data.csv", "a+") as uv_data_file:             uv_data_file.write(str(dt) + ',' + str(uv) +',/n')         time.sleep(10)  if __name__ == '__main__':     main()
Code 8. uv_sensor.py, Python Code to Get and Record UV Sensor Data

The Python code gets the data from Grove UV sensor and writes it to the uv_data.csv, with a timestamp.

Save the code as uv_sensor.py on the UP Squared board. To collect the UV sensor data, run the code:

sudo python uv_sensor.py 
Code 9. Command to Run Python Code

After you’ve collected enough data, press Ctrl+C to stop running code.

Save create_plot.py:

import plotly.plotly as py import plotly.graph_objs as go import pandas as pd from datetime import datetime   # Plotly account authentication py.sign_in('plotly-username', 'plotly-api-key')  # Reading CSV file using Pandas package df = pd.read_csv("uv_data.csv", header=None, parse_dates=True, infer_datetime_format=True, usecols=[0,1])  # Creating a time series with Plotly package data = [go.Scatter(           x=df[0],           y=df[1])] url = py.plot(data)  # Saving URL to a file with open("url_file", "a+") as url_file:     url_file.write(str(url))
Code 10. create_plot.py, Python Code to Create Time Series

Replace plotly-username and plotly-api-key with your Plotly credentials.

This code reads the UV sensor data stored in CSV file and creates a time series that is displayed online. The URL is written to a file, so AWS Lambda can use it later in the tutorial.

Run the code:

sudo python create_plot.py
Code 11. Command to Run Python Code

AWS Lambda

This section shows you how to prepare for and create the AWS Lambda function. This function will read the time series URL from a file and then publish it via MQTT client to the Greengrass’s IoT Cloud. At the end, we will see the MQTT messages received and the time series with UV sensor data.

Go to AWS console and then to the AWS IoT page and select Software from the bottom left. Download the AWS Greengrass Core SDK by clicking on Configure Download. Choose Python* 2.7 and click Download Greengrass Core SDK. After the package has loaded, untar it:

tar –xzvf greengrass-core-python-sdk-1.0.0.tar.gz
Code 12. Command to Untar a Package

Go to the HelloWorld folder:

cd aws_greengrass_core_sdk/examples/HelloWorld
Code 13. Command to Go to the HelloWorld Folder

Unzip the zip file:

unzip greengrassHelloWorld.zip
Code 14. Command to Unzip A Package

Copy url_file to the same folder:

cp <path-to-url-file>/url_file .
Code 15. Command to Copy a File

Copy time_series.py and save it in the same folder:

import greengrasssdk import platform from threading import Timer import time   # Creating a greengrass core sdk client client = greengrasssdk.client('iot-data')  # Retrieving platform information to send from Greengrass Core my_platform = platform.platform()   def time_series_run():     fopen = open("url_file", "r")     url = fopen.read()     if not my_platform:         client.publish(topic='ts/uv', payload='View the time series: {} Sent from Greengrass Core.'.format(url))     else:         client.publish(topic='ts/uv', payload='View the time series: {} Sent from Greengrass Core running on platform'.format(url))      # Asynchronously schedule this function to be run again in 10 seconds         Timer(10, time_series_run).start()   # Start executing the function above time_series_run()  def function_handler(event, context):     return
Code 16. time_series.py, Python Code to Publish Time Series URL to AWS Greengrass

Create a zip file, timeseries.zip, which will be later uploaded to the AWS Lambda function:

zip –r uv_timeseries.zip greengrass_common/ greengrass_ipc_python_sdk/ greengrasssdk/ time_series.py
Code 17. Command to Create a Zip File

Go to AWS console, click Services on top left, put Lambda in search bar and click on it. The Lambda Management Console will open.

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 1. AWS Lambda Functions View

Click Create function.

If not selected, select Author from scratch and fill out needed fields:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 2. AWS Lambda Create Function View

Click Create function.

Upload time_series.zip. Change handler name to time_series.function_handler. Click Save:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 3. Creating AWS Lambda Function View

Click on Actions, select Create new version and call it the first version in description:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 4. Publishing New Version of AWS Lambda Function

Click Publish.

Go to IoT Core/AWS IoT console. Choose Greengrass from leftside menu, select Groups underneath it, and select your group from main window:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 5. AWS Greengrass Groups View

Select Lambda from the leftside menu. Click Add Lambda on right top corner of Lambdas screen:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 6. AWS Greengrass Group View

Select Use Existing Lambda:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 7. Adding AWS Lambda Function for the Greengrass Group

Select time_series from the menu and click Next:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 8. Using Existing AWS Lambda

Choose Version 1 and click Finish:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 9. Selecting Lambda Version

Click on dotted area and select Edit Configuration:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 10. Lambda Functions Within Greengrass Group View

Change Timeout to 25 seconds and choose Lambda lifecycle to be a long-lived function:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 11. Editing Lambda Function View

Click Update on the bottom of the page.

Click the little grey back button, select Subscriptions.

Click Add Subscription or Add your first Subscription:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 12. Adding Subscription View

For the source, choose from Lambdas tab, select time_series. For the target, select IoT Cloud:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 13. Editing Subscription View

Click Next. Add ts/uv for the topic:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 14. Editing Topic for Subscription View

Click Finish.

On the group header, click Actions, select Deploy and wait until it is successfully completed:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 15. Subscriptions View

Go to the AWS IoT console. Select Test from the leftside menu. Type ts/uv in the topic field, change MQTT payload display to display it as strings, and click Subscribe to topic:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 16. MQTT Client View

After some time, messages should display on the bottom of the screen:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 17. MQTT Messages View

Copy the URL and paste it in browser. The time series of UV sensor data is shown below:

Setting Up a Time Series for Sensor Data Using Amazon Web Services (AWS) Greengrass and the UP Squared Board

Figure 18. UV Sensor Data Time Series

About the author

Rozaliya Everstova is a software engineer at Intel in the Software and Services Group working on scale enabling projects for Internet of Things.

More on UP Squared*