Matplotlib, Raspberry Pi, MySQL, RasPiConnect and Project Curacao

January 27, 2014 Matplotlib Graphs, Project Curacao, Python Raspberry Pi Software, Raspberry Pi, RasPiConnect

One of the things we are using the Raspberry Pi on Project Curacao for is to generate graphs.  Yes, We know we could ship the data to another server and generate the graphs there, but since we have a nice linux platform, we decided to make the graphs on site.  Then we ship the graphs both to a webpage and also over to the RasPiConnect directory for display on the RasPiConnect App (www.milocreek.com).

Here’s a picture of the embedded graph (in RasPiConnect):



The RasPiConnect Local.py code for a graph follows:

#W-11 is Pi System Status
if (objectServerID == “W-11”):
                #check for validate request
                if (validate == “YES”):
                        outgoingXMLData += Validate.buildValidateResponse(“YES”)
                        outgoingXMLData += BuildResponse.buildFooter()
                        return outgoingXMLData
            # normal response requested
imageName = “systemstatistics.png”
              responseData = “<html><head>”
                responseData += “<title></title><style>body,html,iframe{margin:0;padding:0;}</style>”
                responseData += “</head>”
                responseData += “<body><img src=””
                responseData += Config.localURL()
                responseData += “static/”
                responseData += imageName
                responseData += “” type=”jpg” width=”800″ height=”300″>”
                responseData +=“</body>”
                responseData += “</html>”
                outgoingXMLData += BuildResponse.buildResponse(responseData)
      outgoingXMLData += BuildResponse.buildFooter()
                return outgoingXMLData

Note that if you want to use a button to select different type of graphs for the same RasPiConnect control, you write the graph name to a file and the read the file to get what graph to display.  Make sure you assign the Web Control to refresh when you push the button.  Then you get an immediate response.  

You can select a different graph by using a feedback control button to write that file (that contains the graph name) and cycle through graphs (see my video on this blog for an example) as below:

# FB-11 – change the graph display
if (objectServerID == “FB-11”):
                #check for validate request
# validate allows RasPiConnect to verify this object is here
                if (validate == “YES”):
                        outgoingXMLData += Validate.buildValidateResponse(“YES”)
                        outgoingXMLData += BuildResponse.buildFooter()
                        return outgoingXMLData
# not validate request, so execute
               responseData = “XXX”
if (objectName is None):
objectName = “XXX”
               lowername = objectName.lower()
                if (lowername == “display voltages”):
                        responseData = “display currents”
                        responseData = responseData.title()
                f = open(“./local/GraphSelect.txt”, “w”)
                f.write(lowername)
                f.close()
                elif (lowername == “display currents”):
                        responseData = “display solar/wind”
                        responseData = responseData.title()
                f = open(“./local/GraphSelect.txt”, “w”)
                f.write(lowername)
                f.close()
                elif (lowername == “display solar/wind”):
                        responseData = “display voltages”
                        responseData = responseData.title()
                f = open(“./local/GraphSelect.txt”, “w”)
                f.write(lowername)
                f.close()
# defaults to display currents
                else:
                        lowername = “display currents”
                f = open(“./local/GraphSelect.txt”, “w”)
                f.write(lowername)
                f.close()
                        responseData = “display voltages”
                        responseData = lowername.title()
                outgoingXMLData += BuildResponse.buildResponse(responseData)
                        outgoingXMLData += BuildResponse.buildFooter()

                return outgoingXMLData

We decided to use the excellent python based matplotlib package.  Here are the steps to get the necessary packages:

$ sudo apt-get install libblas-dev        ## 1-2 minutes
$ sudo apt-get install liblapack-dev      ## 1-2 minutes
$ sudo apt-get install python-dev        ## Optional
$ sudo apt-get install libatlas-base-dev ## Optional speed up execution
$ sudo apt-get install gfortran           ## 2-3 minutes
$ sudo apt-get install python-setuptools  ## ?
$ sudo easy_install scipy                 ## 2-3 hours

$ sudo apt-get install python-matplotlib  ## 1 hour

Thank you https://wyolum.com/numpyscipymatplotlib-on-raspberry-pi/

We run graphs on our pi system based upon entries in apschedule:

    scheduler = Scheduler()
    job = scheduler.add_cron_job(powerdatacollect.datacollect5minutes, minute=”*/5″, args=[‘main’, 0])
    job = scheduler.add_cron_job(watchdogdatacollect.watchdogdatacollect, minute=”*/5″, args=[‘main’, 30])
    job = scheduler.add_cron_job(environdatacollect.environdatacollect, minute=”*/15″, args=[‘main’, 10])
    job = scheduler.add_cron_job(systemstatistics.systemstatistics15minutes, minute=”*/15″, args=[‘main’, 20])
    job = scheduler.add_cron_job(doallgraphs.doallgraphs, minute=”*/15″, args=[‘main’,10,60])
    # camera
    job = scheduler.add_cron_job(useCamera.takeSinglePicture, hour=”*”, args=[‘main’,50])
    # send daily picture
    job = scheduler.add_cron_job(sendPictureEmail.sendPictureEmail, hour=”22″,minute=”20″, args=[‘main’,0])

One good thing about running tasks like this in another thread, is that a hang in one of the threads does not stop the whole system.  Just that thread.


Here is the example matplotlib code and graph for one of the graphs (including the call to MySQL to get the data).  We are using the Solar/Wind line to show which power source is selected.  Reg Wind Volt gives the voltage on the regulated side of the DC/DC 12V – 6V Buck converter (fed to the solar power charging circuitry) and Unreg Wind Volt is on the Wind Turbine side of the converter.  We choose to use just points because of the intermittent nature of the wind.  See the post on the wind storm we had a couple of weeks ago.  This was the graph generated after a couple of trips in the car with various wind turbines.

# solar wind graph generation
# filename: solarwindgraph.py
# Version 1.4 01/12/14
#
# contains graphing routines
#
#
 
import sys
import time
import RPi.GPIO as GPIO
 
import gc
import datetime
 
import matplotlib
# Force matplotlib to not use any Xwindows backend.
matplotlib.use(‘Agg’)
 
from matplotlib import pyplot
from matplotlib import dates
 
import pylab
 
import MySQLdb as mdb
 
sys.path.append(‘/home/pi/ProjectCuracao/main/config’)
 
# if conflocal.py is not found, import default conf.py
 
# Check for user imports
try:
        import conflocal as conf
except ImportError:
        import conf
 
def  solarwindgraph(source,days,delay):
 
 
 
        print(“solarwindgraph source:%s days:%s delay:%i” % (source,days,delay))
        print(“sleeping :”,delay)
        time.sleep(delay)
        print(“solarwindgraph running now”)
 
 
        # blink GPIO LED when it’s run
        GPIO.setmode(GPIO.BOARD)
        GPIO.setup(22, GPIO.OUT)
        GPIO.output(22, False)
        time.sleep(0.5)
        GPIO.output(22, True)
 
        # now we have get the data, stuff it in the graph
 
        try:
                print(“trying database”)
                db = mdb.connect(‘localhost’, ‘root’, conf.databasePassword, ‘ProjectCuracao’);
 
                cursor = db.cursor()
 
                query = “SELECT TimeStamp, RegulatedWindVoltage, UnregulatedWindVoltage, SolarWind FROM batterywatchdogdata where  now() – interval %i hour < TimeStamp” % (days*24)
                cursor.execute(query)

 

                result = cursor.fetchall()
                t = []
                s = []
                u = []
                v = []
 
                for record in result:
                        t.append(record[0])
                        s.append(record[1])
                        u.append(record[2])
                        v.append(record[3])
 
                print (“count of t=”,len(t))
 
                # scale array for Solar =0 Wind = 1
 
                for i in range(len(v)):
                        v[i] = v[i] * 10
 
                #dts = map(datetime.datetime.fromtimestamp, t)
                #print dts
                fds = dates.date2num(t) # converted
                # matplotlib date format object
                hfmt = dates.DateFormatter(‘%m/%d-%H’)
 
                fig = pyplot.figure()
                fig.set_facecolor(‘white’)
                ax = fig.add_subplot(111,axisbg = ‘white’)
                ax.vlines(fds, -200.0, 1000.0,colors=’w’)
 
                ax.xaxis.set_major_locator(dates.HourLocator(interval=6))
                ax.xaxis.set_major_formatter(hfmt)
                ax.set_ylim(bottom = -200.0)
                pyplot.xticks(rotation=’vertical’)
                pyplot.subplots_adjust(bottom=.3)
                pylab.plot(t, s, color=’b’,label=”Reg Wind Volt”,linestyle=””,marker=”.”)
                pylab.plot(t, u, color=’r’,label=”Unreg Wind Volt”,linestyle=””,marker=”.”)
                pylab.plot(t, v, color=’g’,label=”Solar/Wind”,linestyle=”-“,marker=”.”)
                pylab.xlabel(“Hours”)
                pylab.ylabel(“Voltage”)
                pylab.legend(loc=’upper left’)
 
                pylab.axis([min(t), max(t), 0, 20])
                pylab.figtext(.5, .05, (“Solar / Wind System Last %i Days” % days),fontsize=18,ha=’center’)
 
                pylab.grid(True)
 
                pyplot.show()
                pyplot.savefig(“/home/pi/RasPiConnectServer/static/solarwindgraph.png”)
 
        except mdb.Error, e:
 
                print “Error %d: %s” % (e.args[0],e.args[1])
 
        finally:   
 
                cursor.close()
                db.close()
 
                del cursor
                del db
 
                fig.clf()
                pyplot.close()
                pylab.close()
                del t, s, u, v
                gc.collect()
                print(“solarwindgraph finished now”)