Tuesday, 6 November 2012

Logging electricity usage

As mentioned previously, I wanted to be able to log my electricity usage.  I knocked up a circuit with an LDR and my Uno which wrote some data out of it's serial port every time it detected a flash of the LED on my electricity meter.  The LDR was simply attached to the meter with some Blutack.  I had an extra long (5m + a couple of 1m extensions) USB cable running from my kitchen where the meter is located, into my lounge where my laptop usually sits.  On here, was a small Perl program which read the serial port and called a URL which lived on my server upstairs.  This updated a database with various values passed over serial - namely the gap between flashes, and the calculated watt/hours.

This was fine, and worked flawlessly for about 4 weeks, but I was getting tired of not being able to move my laptop around the house without disrupting my monitoring, so I decided to put a stop to it, on a temporary basis.

This was over a month ago!

I bought a basic Arduino setup from Ebay, simply an ATMega328P and a crystal and voltage regulator.  I also purchased a couple of nrf24l01 wireless devices and a poor-mans ethernet shield - the ENC28J60 chip based ethernet controller.  The plan was to run the bare bones Arduino in the electricity meter cupboard and have it talk wirelessly to another Arduino next to my server, and use the ethernet controller to connect to my network.  I have played with the ethernet controller for a couple of days with the jcw library, and although the library does a very good job, it's just not as nice as the offical Wiznet5100 library, so today I splashed out on a W5100 controller, along with two Arduino Nano's.  The bare-bones Arduino will be ideal for the final solution, but the lack of programming interface is a real pain for development.

Once these new parts arrive - probably 2-3 weeks as they are from HK, I will get coding with the Mirf library and the Wiznet library to get something up and running.

Temperature Logging

Not content with the single room temperature I can obtain from my Heatmiser, I have started work on a solution using the DS18b20 one-wire temperature sensors.

These are slightly more expensive than analogue sensors, but they are pre-calibrated and in my opinion much nicer to use.

I bought a pack of 10 from China for under US$9 which is amazing value compared to local options.

Using the simple, but extremely useful details at http://www.hacktronics.com/Tutorials/arduino-1-wire-address-finder.html I was up and running in no time at all.  This site appealed over others as it specifically tells you how to identify each DS18b20 you have on your network, and get it's address - something which most other tutorials miss, and instead just return all sensors, which is useless if you wish to monitor specific ones (and have multiple on the same network).

I didn't have any 4.7K resistors to hand, so paralleled up a couple of 10k's which seem to work fine - infact I think even a 10k would work over short distances from my testing.  I've not bothered with parasitic power.

With this example up and running, I decided to dig out my poor-mans ethernet controller again, and see if I could make it do anything interesting.  With a little head-scratching and some could-be-improved code, I have a little web server running on an Arduino now which reports the current temperature.  

Due to the fact I can't work out out to pass floats with PSTR I ended up multiplying the temperature by 100 to get into an int and then casting the float to an int.  If anyone has a better suggestion I'm all ears.

The code is pretty straight forward, and 90% of it is from the jcw library.



// Present a "Will be back soon web page", as stand-in webserver.
// 2011-01-30 <jc@wippler.nl> http://opensource.org/licenses/mit-license.php
 
#include <EtherCard.h>
#include <OneWire.h>
#include <DallasTemperature.h>
#define STATIC 0  // set to 1 to disable DHCP (adjust myip/gwip values below)
#define ONE_WIRE_BUS 3
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);
DeviceAddress insideThermometer = { 0x28, 0xE8, 0x94, 0x3C, 0x02, 0x00, 0x00, 0x2B };
#if STATIC
// ethernet interface ip address
static byte myip[] = { 192,168,1,200 };
// gateway ip address
static byte gwip[] = { 192,168,1,1 };
#endif

// ethernet mac address - must be unique on your network
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x30,0x31 };

byte Ethernet::buffer[500]; // tcp/ip send and receive buffer

void setup(){
  Serial.begin(57600);
  Serial.println("\n[backSoon]");
  sensors.begin();
  sensors.setResolution(insideThermometer, 10);
  if (ether.begin(sizeof Ethernet::buffer, mymac) == 0) 
    Serial.println( "Failed to access Ethernet controller");
#if STATIC
  ether.staticSetup(myip, gwip);
#else
  if (!ether.dhcpSetup())
    Serial.println("DHCP failed");
#endif

  ether.printIp("IP:  ", ether.myip);
  ether.printIp("GW:  ", ether.gwip);  
  ether.printIp("DNS: ", ether.dnsip);  
}

//byte Ethernet::buffer[500];
BufferFiller bfill;
static word homePage() {
sensors.requestTemperatures();
float tempC = sensors.getTempC(insideThermometer)*100;
Serial.println(tempC);
int tempCi = (int)tempC;

  bfill = ether.tcpOffset();
  bfill.emit_p(PSTR(
    "HTTP/1.0 200 OK\r\n"
    "Content-Type: text/html\r\n"
    "Pragma: no-cache\r\n"
    "\r\n"
    "<title>RBBB server</title>" 
    "<h1>Temp: $D</h1>"),
      tempCi);
  return bfill.position();
}



void printTemperature(DeviceAddress deviceAddress)
{
  float tempC = sensors.getTempC(deviceAddress);
  if (tempC == -127.00) {
    Serial.print("Error getting temperature");
  } else {
    Serial.print(tempC);
  }
}



void loop(){
    Serial.print("Getting temperatures...\n\r");
  sensors.requestTemperatures();
  
  Serial.print("Inside temperature is: ");
  printTemperature(insideThermometer);
  Serial.print("\n\r");
  
  
  word len = ether.packetReceive();
  word pos = ether.packetLoop(len);
  
  if (pos)  // check if valid tcp data is received
    ether.httpServerReply(homePage()); // send web page data
}


Following my experiments I decided to approach this from a different way.  I now run a webclient on the Arduino which fetches a PHP page every so often with a bunch of temperatures in the query string.  The code for this is below:


#include <EtherCard.h>

#include <OneWire.h>
#include <DallasTemperature.h>
#define STATIC 0  // set to 1 to disable DHCP (adjust myip/gwip values below)
#define ONE_WIRE_BUS 3

OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

DeviceAddress Office = { 0x28, 0xE8, 0x94, 0x3C, 0x02, 0x00, 0x00, 0x2B };
DeviceAddress Outside = { 0x28, 0xE8, 0x94, 0x3C, 0x02, 0x00, 0x00, 0x2B };
DeviceAddress Master = { 0x28, 0xE8, 0x94, 0x3C, 0x02, 0x00, 0x00, 0x2B };
DeviceAddress Howard = { 0x28, 0xE8, 0x94, 0x3C, 0x02, 0x00, 0x00, 0x2B };

byte mymac[] = { 0x74,0x69,0x69,0x2D,0x30,0x31 };

char website[] PROGMEM = "<hostname of remote server>";
uint32_t timer;
byte Ethernet::buffer[700];
Stash stash;
char buffer[25];

void setup ()
{
    Serial.begin(57600);
    Serial.println("\n[DS18b20 HTTP Updater]");
    
    sensors.begin();
    sensors.setResolution(Office, 12);
    sensors.setResolution(Outside, 12);
    sensors.setResolution(Master, 12);
    sensors.setResolution(Howard, 12);
    
    if (ether.begin(sizeof Ethernet::buffer, mymac) == 0) 
    Serial.println( "Failed to access Ethernet controller");
    
    if (!ether.dhcpSetup())
    Serial.println("DHCP failed");
    
    ether.printIp("IP:  ", ether.myip);
    ether.printIp("GW:  ", ether.gwip);  
    ether.printIp("DNS: ", ether.dnsip);  

    Serial.println("Please wait.  Resolving DNS...");
    if (!ether.dnsLookup(website))
    Serial.println("DNS failed");
    
    ether.printIp("DNS Resolved...Server IP: ", ether.hisip);
}

void loop () {
ether.packetLoop(ether.packetReceive());
  if (millis() > timer) 
  {
    timer = millis() + 30000;
    sensors.requestTemperatures();
    float tempOffice  = sensors.getTempC(Office);
    float tempOutside = sensors.getTempC(Outside);
    float tempMaster  = sensors.getTempC(Master);
    float tempHoward  = sensors.getTempC(Howard);
    updateTemp(tempOffice, tempOutside, tempMaster, tempHoward);
  }
}


static void updateTemp ( float tempOffice, float tempOutside, float tempMaster, float tempHoward ) 
{
 byte sd = stash.create();
 stash.print("GET /heatmiser/update_temp.php?");
        stash.print("Office=");
        stash.print(tempOffice);
        stash.print("&Outside=");
        stash.print(tempOutside);
        stash.print("&Master=");
        stash.print(tempMaster);
        stash.print("&Howard=");
        stash.print(tempHoward);
        stash.print(" HTTP/1.0 \r\n");
 stash.println("\r\n");
 stash.save();
 Stash::prepare(PSTR("$H"), sd);
 ether.tcpSend();
}

There is some PHP on my server which decodes the query string and adds the data to the database.  Currently each of the 4 sensors actually reference the same ds19b20 but that's just for testing, I have a bag full of them I plan to add over the coming days around the house.

Downside to this code is that it requires each sensor to be hardcoded, but that gives them friendly names rather than just index ID's which would mean you didn't know which sensor was which.

Logging Heatmiser Data

After getting my Heatmiser thermostat, I wasn't content with just using the (rather 2000's style) web GUI, I wanted a way to interact with the device directly, and at the time there was no Android app for Heatmiser.

With the help of viewing lots of source code from the Heatmiser web GUI and traces from wireshark, I was able to identify the basics of how the forms on the web GUI work.  I know there is the iPhone interface, but it's a pretty evil protocol for what is a very small basic set of functions I needed, and I decided that interacting with the web GUI was far quicker and easier.

The code was/is less than optimal or pretty, but it works (just about).

The code to log the current and set temperature along with the heating status is Perl based, and written to run in a unix environment.

#!/usr/bin/perl
use DBI;

$dbh = DBI->connect('DBI:mysql:Monitoring', '<username>', '<password>') || die "Could not connect to database: $DBI::errstr";

@userinput = `wget -q -O- http://<heatmiser_IP>/right.htm`;

foreach (@userinput) {
$line = $_;
if ( ( $cur,$set,$state ) = $line =~ m/.*Actual.*'5'>(\d{1,}.*?)&nbsp.*'4'>(\d{1,}.*?)&nbsp.*((ON|OFF))/ )
{
$date = `date "+%Y-%m-%d %H:%M"`;
if ( $state =~ /OFF/ )
{
$state = "0";
}
elsif ( $state =~ /ON/ )
{
$state = "1";
}
chomp($date);
$date = "$date:00"; //Fix the date to be 00 seconds
$dbh->do("INSERT INTO `heating` SET date=\'$date\',actualtemp=$cur,settemp=$set,onoff=$state");
}
}


And the PHP to make the Google charts work:

<?php
$DB_NAME 
'Monitoring';$DB_HOST '<HOST>';$DB_USER '<USERNAME>';$DB_PASS '<PASSWORD>';
$mysqli = new mysqli($DB_HOST$DB_USER$DB_PASS$DB_NAME);
if (
mysqli_connect_errno()) {
    
printf("Connect failed: %s\n"mysqli_connect_error());
    exit();
}
if(!isset(
$_REQUEST['duration']))
{
    
$duration 1;
}
else{
    
$duration $_REQUEST['duration'];
}
$date date("Y-m-d H:i:s",time()-($duration*86400));$query "SELECT * FROM `heating` WHERE `date` > '$date'";$result $mysqli->query($query) or die($mysqli->error.__LINE__);?>
<html>
  <head>
    <script type="text/javascript" src="https://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:["corechart"]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('string', 'Date');
        data.addColumn('number', 'Actual Temp');
        data.addColumn('number', 'Set Temp');
        data.addColumn('number', 'State (1=on, 0=off)');
        data.addRows([
    <?phpwhile($row $result->fetch_assoc()) {
    echo 
"['";
    echo 
$row['date'];
    echo 
"', ";
    echo 
$row['actualtemp'];
    echo 
", ";
    echo 
$row['settemp'];
    echo 
", ";
    echo 
$row['onoff'];
    echo 
"],\n";
    
$date $row['date'];
    
$actualtemp $row['actualtemp'];
    
$settemp $row['settemp'];
    
$onoff =$row['onoff'];
    }
        echo 
"['";
        echo 
$date;
        echo 
"', ";
        echo 
$actualtemp;
        echo 
", ";
        echo 
$settemp;
        echo 
", ";
        echo 
$onoff;
        echo 
"]\n";

    
mysqli_close($mysqli);?>        ]);

        var options = {
        title: 'Temperature History',
        vAxis: {title: "Temperature"},
        hAxis: {title: "Date / Time"},
        };

        var chart = new google.visualization.AreaChart(document.getElementById('chart_div'));
        chart.draw(data, options);
      }
    </script>
  </head>
  <body>

    <div id="chart_div" style="width: 1000; height: 640;"></div>
  </body>
</html>

-------------

This produces a chart such as this, which shows a 24hour period.  To display 2 days, pass "?duration=2" to the code, or for 12 hours, pass "?duration=0.5", if you want to display any other values, it is just a factor of 1 = where 1 = 24hours, so 1hour = 0.0417.


The Beginnings

I've had an Arduino for the best part of a year now.  I originally bought the Uno for a project I had in mind, but one thing lead to another, and it got put on the back shelf.

Then in February 2012 I moved into a new house, needing to buy a new central heating thermostat, and being somewhat of a geek (the wife calls me a nerd) I decided to invest in a WiFi enabled Heatmiser thermostat with the idea of tracking my heating usage.  For the past 7 months or so, I have been logging my energy usage using Google Charts and it has been a rather significant eye opener into how much gas I use along with the temperature fluctuations in my house.

This was fine for a few months, but I was lacking electricity monitoring and temperature else where - such as outside.

I could have bought a standalone energy monitor, but where is the fun in that?  I decided to investigate using the LED indicator on my meter which flashes once per Watt Hour used - ie 1,000 flashes per KWh.  This took me into my first "real" Arduino project.  More to follow.