ENS Environment Notification System

Update: I’ve moved the server portion of this system to Elasticsearch/Kibana. No Offsite data stored.

The ENS is designed for monitoring Temperature/Humidity/Light/etc. in the environment around your land, home, or outbuildings. The system consists of one or more sensor nodes and a Server to collect and manage the data.

The aim is to prevent any losses due to Overheated Greenhouse or Failed Refrigerators/Freezers. The sensors can also be configured with Humidity, Soil Moisture, Light; and they can be powered with AC or a DC solar system. With some inexpensive wireless technology, we can span miles between sensors(Line of sight) if this is required.

I am selling this as a package to folks and businesses around Saint Louis. If you would like to build you own, I can provide all the code and remote assistance for $60 per hour (Onsite, Minimum 2 hours.)

One Sensor and Server starts at $1300, with $300/year maintenance (Backups, Software, and Hardware replacement included).

Additional Sensors start at $130 + $30 per year.

If this is something you are interested in, send a message my way: ryan at 2matoes.com

Here are the guts of the system:

Sense

Raspberry Pi sensor

Raspberry Pi

Start off with argparse This makes it easy to add cron jobs for different sensors.

Sense the data from the sensor, format it and send it to the logstash server.


import argparse,socket

#python writeTempsToRedisServer8090.py --site "SlowDownHome" --sensor "outdoor" --sensorsystem "28-000006152042" --server "192.168.1.10"

argparser = argparse.ArgumentParser(description='Arguments that may be parsed.',epilog="The Epilog")
argparser.add_argument('--site',type=str,required=True,help='Site the Device is Located. Required.')
argparser.add_argument('--sensor',type=str,required=True,help='Sensor Real Name. i.e. "Outdoor"')
argparser.add_argument('--sensorsystem',type=str,required=True,help='Sensor Real Name. i.e. "209710825416"')
argparser.add_argument('--sensortype',type=str,required=True,help='Sensor Type(Temp,Humidity,Soil Temp,Soil Hydration,etc.): T,H,ST,SH')
argparser.add_argument('--server',type=str,required=True,help='Server IP address or hostname')
argparser.add_argument('--port',default=8090,type=int,help='Port number on the server')
args = argparser.parse_args()

def GetSensorData(sensorsystem):
  try:
    TemperatureReadFromSensor = open("/sys/bus/w1/devices/"+sensorsystem+"/w1_slave")
  except Exception, e:
    print "Unable to open sensor File."
    print e
  else:
    sensordata = ReturnC(TemperatureReadFromSensor.read())
    TemperatureReadFromSensor.close()
  finally:
    return sensordata

def main():
  sensordata = GetSensorData(args.sensorsystem)
  nodeoutput = args.site+";"+args.sensor+";"+args.sensorsystem+";"+args.sensortype+","+str(sensordata)
  SendSensorDataToRedisServer8090(nodeoutput)

def ReturnC(text):
  secondline = text.split("\n")[1]
  temperaturedata = secondline.split(" ")[9]
  temperature = float(temperaturedata[2:])
  ctemp = temperature / 1000
  #print "Celsius: "+str(ctemp)
  return ctemp

def ReturnFfromC(ctemp):
  ftemp=ctemp*1.8+32
  #print "Fahrenheit: " +str(ftemp)
  return ftemp

def SendSensorDataToRedisServer8090(sensordata):
  #Sensor and Temp should look like sensor1,55.43
  print(sensordata)
  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.connect((args.server, args.port))
  s.send(sensordata)
  s.close()

if __name__ == "__main__":
  main()

Arduino sensor

Arduino or Pinoccio

The arduino works similarly. I currently have one running in my greenhouse. But I have stopped using them, favoring the raspberry pi.

A few things happened that made me more interested in moving to raspberry pi only. First off I had problems with compiling, hours and hours of compiling issues to get one sensor working(a couple times it was the version of the arduino IDE… This was mighty frustrating. It would compile, install, but the software would not work.). Then I had everything soldered up to an arduino sensor and then it would not accept my program.

The pi is not perfect; It uses a fair bit more power and theoretically takes longer to configure… but it just works well once set up, and allows for easier Remote configuration.

Another hardware option for the monitor node is the pinoccio; similar to an arduino, yet it has a simpler programming language (think python) and programmable over the air, built in temp sensor, onboard mesh network 802.15.4, onboard lipo battery, and some incredible power saving features(in theory you could run for a couple years, taking a data reading hourly, off of one battery charge). The list of great features goes on… The cost goes up per sensor, but the feature set warrants the price.

Acquire

Initially I went with some Python scripts that would drop the raw data into the database, but I wanted something a bit more resilient.

Logstash

Logstash fit that want. It is fast and can easily validate the data.

input {
  tcp{
    type=>"rawsensordata"
    port=>8090
  }
  tcp{
    type=>"hourly"
    port=>8091
  }
}
filter{
  if [type] == "rawsensordata"{
    grok{
      match => { "message" => "%{DATA:site};%{DATA:sensor};%{DATA:sensorid};%{DATA:sensortype},%{NUMBER:sensordata}" }
    }
    grok{
      match => {"@timestamp" => "%{DATA:date}T%{DATA:utc}Z"}
    }
  }

  if [type] == "hourly"{
    grok{
      match => { "message" => "%{DATA:site};%{DATA:sensor};%{DATA:date};%{NUMBER:hour},%{NUMBER:sensordata}" }
    }
  }
}
output{
  if [type] == "hourly"{
    redis{
      data_type=>"list"
      key=>"hourly:%{site}:%{sensor}:%{date}:%{hour}"
      password=>"n8JylbbtPkBpOlgD"
    }
  }
  if [type] == "rawsensordata"{
    redis{
      data_type=>"list"
      key=>"logstash:%{site}:%{sensor}:%{date}"
      password=>"n8JylbbtPkBpOlgD"
    }
  }
  stdout {
    codec => rubydebug
  }
}

Store

Redis

For storing data I went with Redis, Super fast, simple, and runs on nearly any linux distribution.

Fiddle

Here is where the magic happens…

Scripts to average and upload the data

There is a script that runs hourly to average the past hour of data. This uploads data to the Google API to log data and make the pretty charts mentioned below.

There will be a script that runs every 5 minutes to gather data for the notification system, Nagios.

Notify

Nagios

Nagios is relatively easy to setup and works with just about any notification method that you can script.

Nagios will read from the Redis database and compare the numbers it receives with the warning and critical threshold specified in the setup.

View

Here is the good stuff.

Google Charts

These charts display 3 days/months of data from my Test.

P.S. This is live data

The Code for gspread

import gspread

sheetname = "ENS-TestSheet"
json_key = json.load(open('/home/user/.ssh/google-api-key.json'))
scope = ['https://spreadsheets.google.com/feeds']
credentials = SignedJwtAssertionCredentials(json_key['client_email'], json_key['private_key'].encode(), scope)
gc = gspread.authorize(credentials)
sheet = gc.open(sheetname)


def SendAveragedDataToGoogleSheets(site,sensor,operationdate,databyhour):
  worksheet = None
  try:
    worksheet = sheet.worksheet(site)
  except Exception, e:
    rows = 1
    cols = 3
    worksheet = sheet.add_worksheet(site, rows, cols)
    row = ["sensor","datetime","sensordata"]
    worksheet.insert_row(row,1)
  finally:
    if worksheet == None:
      print "Failure"
  for hour in databyhour:
    row = [sensor,operationdate+"T"+str(hour).zfill(2)+":"+"00Z",str(databyhour[hour])]
    worksheet.append_row(row)