OpenNMS Eclipse IoT Challenge 2017/18 Final Report

This post provides an update to our previous post and video describing The OpenNMS entry to the Eclipse IoT challenge 2017



Project Description

The aim of this project is to use IoT pollution measurements as a compelling use case to prove a design for an MQTT plugin for OpenNMS.
This will position OpenNMS as a highly capable platform for distributed IoT monitoring applications. IoT projects will immediately be able to leverage OpenNMS’s existing very powerful event correlation and performance data visualization in the IoT space. Further more, OpenNMS can be used both collect the IoT data and also to manage the IoT infrastructure itself.
The MQTT plugin allows OpenNMS to connect to multiple MQTT brokers and parse any compressed or uncompressed JSON, CSV, XML (and in the future Protobuf) MQTT payload into OpenNMS events or OpenNMS Performance data. Example configurations are provided for several MQTT data sources including Eclipse Kura.
Performance data can either be stored in OpenNMS’s internal RRD files or alternatively stored in Cassandra using the OpenNMS Newts technology which offers a highly scalable cloud storage solution.
Similarly events can be processed and stored internally in OpenNMS’s Postgres database or forwarded to Elastic Search for a large scale data archive.

A complete power-point presentation of the project which serves as a detailed report is available at Airwatch I0T Challenge Report March 2018

The code associated with this project is available on github at opennms mqtt plugin

Open technology and standards

This project seeks to support the adoption of open technologies and standards for IoT projects by providing an alternative Open Source solution to the cloud hosted IoT environments offered by Google, Amazon , Microsoft etc. It seeks to compliment the work of the Eclipse IoT project and in particular to support the deployment of solutions built on Eclipse Kura. OpenNMS is an open source network and infrastructure management platform written in Java and distributed under the  Affero General Public License (AGPL). Commercial support is available from the OpenNMS Group. OpenNMS supports and/or implements or integrates with the following standards and technologies

  • Karaf OSGi platform
  • Cassandra
  • Elastic Search
  • Postgresql
  • Grafana
  • Kibana
  • ActiveMQ
  • Jboss Rules (Drools)
  • Management protocols supported include SNMP, WMI, JMX, SYSLOG and now MQTT
  • All external integration can be achieved using ReST XML or Json calls.
  • Eclipse is the recommended IDE for working on OpenNMS code.

Applicability of a solution to a specific industry

The specific use case for this implementation has been in support of an emerging project called Solent Airwatch. This project seeks to support  citizen science / crowd sourced data collection of air pollution measurements across a city. Although we are initially targeting the measurement of Air Quality, the same technology could be applied for any other IoT measurement application.

Key to our approach is the leverageing of an existing  data collection platform which has been proven in large scale data collection and processing of network and data centre statistics. Many of the large telecoms players who already use OpenNMS are actively investigating business opportunities for IoT. So this allows them to extend rather than replace an existing platform. In addition, the solution allows them to manage the IoT infrastructure as well as collect data using OpenNMS

Lessons learned

Participating in this IoT challenge has helped us accelerate our development of the Solent Airwatch solution. It has allowed us to learn about the practicalities of implementing an MQTT based solution. Specifically we have learned;

1. Core data collection and connectivity is only part of the problem of Citizen Science. We need to be able to integrate back end data collection with front end web enabled communities to deliver a complete solution. We are in the process of designing a front end web site which will explain and excite interest in the measurements begin collected.

2. The Data science around measurements collection and analysis is very important. We need to be able to support complex cross correlation between data sets in order to leverage the data we are collecting from our IoT infrastructure. This is more complex and less structured than the data we normally deal with from networks or data centres. We are presently investigating integrations with Cassandra and OpenNMS which can support more complex data analytic s using R or Matlab.

3. We had hoped to re-implement our initial Raspberry Pi based Sniffy in Eclipse Kura. We still intend to do this but the documentation and example code surrounding Kura is incomplete and complex. We think the platform will need to become much more accessible if it is to get wider adoption.

We are hoping to get several Master’s students to work on taking this initial proof of concept forwards.

Overall the project has been a very interesting challenge and we will certainly want to be involved with the Eclipse IoT team in the future.