Sensorweb 1.x
2019-09-21
2021-01-07

Sensorweb 1.x

Overview

Sensorweb is a collection of data aggregation software and sensors. It is developed in "C". It offers the possibility of expanding the available sensor list by using a library of functions and programing our own.


There are 4 types of components:

  1. The sensors:
    • network activity (sensorweb_network),
    • cpu temperature (sensorweb_cputemp2),
    • sunshine monitor (sensorweb_sunshine2),
    • air temperature, humidity and barometric pressure (thermo2)
    • miscellaneous ... (see below, sensorweb_oneshot).
    • The last three are in the same module because the data is made available by the same chip sensor (BME280).
  2. The local aggregator (sensorweb_client). This aggregates all the sensors on the same platform and sends it to the server (sensorweb_server)
  3. The server receives the data from the aggregators and databases it.
  4. The sensors, aggregator and server make use of a library that is installed for use as a runtime library and a resource for those wishing to build there own components.

It goes without saying that there can be multiple aggregated feeds per server.

 

Component communication

The communication between the sensors and the local aggregator uses Unix_sockets. The communication between the local aggregators and the server uses TCP/UDP networking. All communication between components are non-blocking.

Forwarding safeguards

To prevent loss of data, the sensors will stack on a storage device the data should the local aggregator not be available.

In the same fashion the local aggregators will stack the data should the server not be available.

The end game of the sensorweb is to store a data in a database. Should the server not be able to store the data to the database, the date will be stacked on disk. All stacks will be forwarded when the faulty components resume.

The communication between components is further protected through an acknowledgement mechanism that ensures proper down-line reception and avoids duplication.

During transmission, a sequence number is added to the data record in order to differentiate between two readings that could have been taken on the same second, same sensor same host. This sequence number is otherwise ignored and finally stripped before storage in the database.

The Data

From the host and its sensors, to the database, phase one, temporary storage

The data from the sensors which is then forwarded by the aggregator follows a very simple layout. It consists of:

  1. A timestamp (unix timestamp, seconds since epoch). This is timezone independent.
  2. A source identifier usually the name of the computer on which runs the sensor
  3. A sensor identifier
  4. A numeric value

When the server receives this data from the aggregator, the server saves it under the following schema:

CREATE TABLE `Data` ( `ID` int(11) NOT NULL AUTO_INCREMENT, `Source` varchar(32) DEFAULT NULL, `Sensor` varchar(32) DEFAULT NULL, `Timestamp` int(11) DEFAULT NULL, `Value` float DEFAULT NULL, PRIMARY KEY (`ID`), KEY `I_Time` (`Timestamp`) ) ENGINE=InnoDB AUTO_INCREMENT=656196 DEFAULT CHARSET=utf8mb4;

Post processing, phase 2, permanent storage

It is clear that keeping the data in this fashion for sensors that report often (in my case, every minute), the data could become huge quiet rapidly especially since in my case I deployed many sensors on a few platforms. To reduce the size of the database this data is moved from the phase one storage to the following schema and storage:

CREATE TABLE `DataByHour` ( `ID` int(11) NOT NULL AUTO_INCREMENT, `Source` varchar(32) DEFAULT NULL, `Sensor` varchar(32) DEFAULT NULL, `Timestamp` int(11) DEFAULT NULL, `Min` float DEFAULT NULL, `Value` float DEFAULT NULL, `Max` float DEFAULT NULL, PRIMARY KEY (`ID`), KEY `I_Time` (`Timestamp`) ) ENGINE=InnoDB AUTO_INCREMENT=139690 DEFAULT CHARSET=utf8mb4;

The two new fields, "Min" and "Max" will hold the respectively the smallest and largest values sensed during the aggregation period (1 hour) while the "Value" field will host the mean of the values. This is simply accomplished via mysql using this cron job executed every hour (file: "/usr/share/sensorweb/Integrate_Raw_Data.sh")

#!/bin/sh mysql -u xxxxxx -pcollect yyyyy << eot select unix_timestamp() into @limit; select (@limit-(@limit%3600)) into @limit; insert into DataByHour SELECT NULL, Source, Sensor, (Timestamp-(Timestamp%3600)) AS T, MIN(Value), AVG(Value), MAX(value) FROM Data WHERE Timestamp<@limit GROUP BY Source, Sensor, T; delete from Data where Timestamp<@limit; eot

Notes: This procedure will aggregate and move the data that belongs to the previous hours. The data of the current hour is left in the first phase storage.

A typical application

A typical application of the use of the stored data is to plot the changes of its value over time.

the following procedure will extract the data required to plot a "Sensor" from the "Source" in the past hours:

#!/bin/sh echo "select Source,Sensor,Timestamp,Min,Value,Max from DataByHour where Timestamp>=(unix_timestamp()-"$1"*24*3600) and Source='"$2"' and Sensor='"$3"' order by Timestamp; select Source,Sensor,Timestamp,Value,Value,Value from Data where Source='"$2"' and Sensor='"$3"' order by Timestamp ;" | mysql -N -u read -pread sensordataNotes: The Data of both first and second phase storage are combined without any redundancy.

The following "gnuplot" script uses this procedures to accomplish the plot:

Copy to Clipboard

Notes: The values for "days", "sensor", and "source" of this script are either to be set passed as parameters. This script adds the timezone offset to display the local time on the plot.

A special case: sensorweb_oneshot

This sensor is meant to collect events rather than continuous data. It is meant to collect data for source that either can not be monitored by a daemon such as the other sensors or where the daemon is overkill. For instance if you wanted to monitor the restart of the sensor platform, you would place a call to the "sensorweb_oneshot" within the "boot" procedure. Alternately you could configure "systemd/sytemctl" to take care of the call.

sensorweb_oneshot [-h] [-d] [-n ] -e [-s ] -v

where:

  • -h will output a quick usage reminder such as above,
  • -d will trigger debugging outputs,
  • -n specifies the name of the originating host,
  • -e is the event name,
  • -s the name of the stack file that will receive the data and
  • -v the value associated with the event.

In this particular case, the "-s" parameter is critical. In most application, the file name given here should be the same as the stack file associated with "sensorweb_client". In such a configuration, when a call to one "sensorweb_oneshot" is done, the data will be placed on the stack and sent to the "sensorweb_server". If not configured in this fashion, the user will be required to see to the processing of the stack file.

Sensorweb configuration file

The different components of sensorweb can be configured when invoked via parameters on the call to the program or values for the parameters can be set in "/etc/default/sensorweb".

This file is executable and its purpose is to set environment variables that will be picked up by the components. For example:

Copy to Clipboard

Raspberry PI installation

Sensorweb has been successfully installed on Raspberry PI Zero through 3B with the Symbian OS. It was also installed as source on Debian ( X86-64 ) and Fedora (X86-64) without requiring adaptations.

Sensorweb has been packaged for distribution and installation via the "Debian" standard tool apt-get.

At the time of this publication, no distribution node has been created, therefor the following:

Get the package and install it
# wget http://.../sensorweb-0.xyz.deb
# apt install ./sensorweb-0.xyz.deb

Create the mysql database and tables

The file "/usr/share/sensorweb/schema.sql" contains the schema of the tables. The database name is "sensordata".

Update the "/etc/default/sensorweb"

Remember to leave this file as belonging to root and with permissions 0x700 or rwx------.

Enable the components you need and start them

Where '...' could be either of "sensorweb_server", "sensorweb_client", "sensorweb_cputemp2", "sensorweb_thermo2" and "sensorweb_network".
# systemctl enable ...
# systemctl start ...

Building your own sensor

The simplest sensor is sensorweb_cputemp2. It reads the file where the CPU temperature is stored, converts the number and forwards the data

Copy to Clipboard

Disclaimer

Use at your own risk. I decline any responsibility pertaining to this web page and the use of the material that it contains.