This post provides an update to our previous post and video describing The OpenNMS entry to the Eclipse IoT challenge 2017

Project Description

The aim of this project is to use IoT pollution measurements as a compelling use case to prove a design for an MQTT plugin for OpenNMS.
This will position OpenNMS as a highly capable platform for distributed IoT monitoring applications. IoT projects will immediately be able to leverage OpenNMS’s existing very powerful event correlation and performance data visualization in the IoT space. Further more, OpenNMS can be used both collect the IoT data and also to manage the IoT infrastructure itself.
The MQTT plugin allows OpenNMS to connect to multiple MQTT brokers and parse any compressed or uncompressed JSON, CSV, XML (and in the future Protobuf) MQTT payload into OpenNMS events or OpenNMS Performance data. Example configurations are provided for several MQTT data sources including Eclipse Kura.
Performance data can either be stored in OpenNMS’s internal RRD files or alternatively stored in Cassandra using the OpenNMS Newts technology which offers a highly scalable cloud storage solution.
Similarly events can be processed and stored internally in OpenNMS’s Postgres database or forwarded to Elastic Search for a large scale data archive.

A complete power-point presentation of the project which serves as a detailed report is available at Airwatch I0T Challenge Report March 2018

The code associated with this project is available on github at opennms mqtt plugin

Open technology and standards

This project seeks to support the adoption of open technologies and standards for IoT projects by providing an alternative Open Source solution to the cloud hosted IoT environments offered by Google, Amazon , Microsoft etc. It seeks to compliment the work of the Eclipse IoT project and in particular to support the deployment of solutions built on Eclipse Kura. OpenNMS is an open source network and infrastructure management platform written in Java and distributed under the  Affero General Public License (AGPL). Commercial support is available from the OpenNMS Group. OpenNMS supports and/or implements or integrates with the following standards and technologies

• Karaf OSGi platform
• Cassandra
• Elastic Search
• Postgresql
• Grafana
• Kibana
• ActiveMQ
• Jboss Rules (Drools)
• Management protocols supported include SNMP, WMI, JMX, SYSLOG and now MQTT
• All external integration can be achieved using ReST XML or Json calls.
• Eclipse is the recommended IDE for working on OpenNMS code.

Applicability of a solution to a specific industry

The specific use case for this implementation has been in support of an emerging project called Solent Airwatch. This project seeks to support  citizen science / crowd sourced data collection of air pollution measurements across a city. Although we are initially targeting the measurement of Air Quality, the same technology could be applied for any other IoT measurement application.

Key to our approach is the leverageing of an existing  data collection platform which has been proven in large scale data collection and processing of network and data centre statistics. Many of the large telecoms players who already use OpenNMS are actively investigating business opportunities for IoT. So this allows them to extend rather than replace an existing platform. In addition, the solution allows them to manage the IoT infrastructure as well as collect data using OpenNMS

Lessons learned

Participating in this IoT challenge has helped us accelerate our development of the Solent Airwatch solution. It has allowed us to learn about the practicalities of implementing an MQTT based solution. Specifically we have learned;

1. Core data collection and connectivity is only part of the problem of Citizen Science. We need to be able to integrate back end data collection with front end web enabled communities to deliver a complete solution. We are in the process of designing a front end web site which will explain and excite interest in the measurements begin collected.

2. The Data science around measurements collection and analysis is very important. We need to be able to support complex cross correlation between data sets in order to leverage the data we are collecting from our IoT infrastructure. This is more complex and less structured than the data we normally deal with from networks or data centres. We are presently investigating integrations with Cassandra and OpenNMS which can support more complex data analytic s using R or Matlab.

3. We had hoped to re-implement our initial Raspberry Pi based Sniffy in Eclipse Kura. We still intend to do this but the documentation and example code surrounding Kura is incomplete and complex. We think the platform will need to become much more accessible if it is to get wider adoption.

We are hoping to get several Master’s students to work on taking this initial proof of concept forwards.

Overall the project has been a very interesting challenge and we will certainly want to be involved with the Eclipse IoT team in the future.

In December Ronny, David and I went to TM forum Digital disruption in San Jose. OpenNMS has been involved in the TM Forum for many years but this was the first time we had taken a stand at the show. We have always had a bit of a love/hate relationship with the TM Forum because from a business perspective it is hard to justify the membership based upon the natural demographic of our customers.

The TM Forum  have traditionally only addressed the needs of Telecoms service providers but as the industry has been converging, they have realised the need to expand their membership to included wider IT/Web industry players.

This change in focus is potentially very good for OpenNMS since we have made a lot of headway in the wider IT space but have found it difficult to penetrate Telecoms. As part of their efforts to  consciously move from the Telco space into the wider Cloud/IT/Web infrastructure providers market the TM Forum have moved their North American show to Silicon Valley in order to attract this audience. We wanted to use the show to demonstrate the new STUI display and features of OpenNMS 14 which were particularly interesting to a cloud provider.

Evaluation

We have quite a large number of fairly senior decision makers to the stand but unfortunately the worst storm to hit San Francisco in 4 years happened half through the week and most of the delegates left early !!. Overall I think the show was worthwhile but given the footfall on the stand, I think the project could have had better value for the money we spent. In future I think if we continue to engage with the TM Forum the most cost effective way will be to participate in catalysts rather than take a stand. Catalysts are real projects in which we can demonstrate OpenNMS technology along side other vendors equipment and systems. I think overall it is a better window for OpenNMS than just a stand.

STUI

If you want to know more about STUI, see the youtube video which David Hustace has created here.

http://www.youtube.com/watch?v=2oudyfiwijw

Marketing OpenNMS to Government:- 

Traditionally Open Source projects don’t have much of a marketing budget and tend to rely on world of mouth or technical conferences to promote the value of the product. This has served OpenNMS well in the past but we are coming to think that the reason many technically inferior closed source products do so well is because of their ability to market to decision makers in the ICT sector. With this in mind we have been putting a lot more effort into getting the message out about OpenNMS this year through workshops, attendance at shows and direct marketing. Of course the problem is, as John Wanamaker is quoted as saying “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”

One of the most difficult areas to crack is Government which is naturally risk averse and in the present climate finds it difficult to budget for new innovations. However, we decided this year to reach out directly through SOCTIM the trade association which represents the ICT managers in the UK Government . They run an annual trade show and this year we teamed up with Shadowcat Ltd to rent a stand.

Our Enthusiastic OpenNMS reps on Day 2

 Our Enthusiastic OpenNMS reps on Day 1(Ian Norton and Craig Gallen)

What is SOCTIM ?

SOCTIM is the professional body for people involved in the leadership and management of IT and digitally enabled services delivered for public benefit. Each year SOCTIM organises a conference for IT professionals in the UK public sector. Drawing on contributions from leading politicians, industry experts and local practitioners, Socitm 2014 provided an exciting opportunity for ICT professionals from across the public sector to take on the challenge of designing and articulating the value of a deeper digital presence.

http://www.socitmconference.com

How useful was the Show?

We wanted to give as wide a view as possible of the potential for OpenNMS because for people who have never used a network management solution, we have to spark the imagination and show them what is possible. We demonstrated the key new features of OpenNMS V 1.14 but we also demonstrated how OpenNMS can be used to build solutions such as the Open Alert App we prototyped at Jersey Digital Hub .

In addition we showed how services can be built using OpenNMS. Shadowcat have built a subscription management offering on top of OpenNMS called ShadowNMS which uses OpenNMS as the core engine but provides a much simpler user experience for subscribers who want to manage a small number of servers or sites using a cloud solution. (We will review this separately in another blog) .

We did attract a lot of interest but it was obvious that the usual Fear Uncertainty and Doubt (FUD) with respect to Open Source still appeared to pervade the community. We got the usual scepticism about ‘free software’ which usually began to a bait once we told the OpenNMS story.  There is a general unwillingness to engage with open source in government because of the perceived risk and because many don’t understand the Open Source business model. At one session, a local council was promoting a suit of finance software they had written with a view to selling it  as a COTS solution to other local authorities. I asked the obvious question, ‘why didn’t you open source the solution for other authorities’?  but the response indicated that they felt this would be poor use of taxpayers money.  It appears to me that if the objective was to reduce the cost to the local councils for maintaining a suit of software, the open source solution would be much better use of tax payers money than trying to commercialise the IPR in a limited market. Cest la Vie…

Cost effective Innovation also featured highly in the conference agenda. However not much thought was given to how agile development is actually done and the value of open source development techniques in the process of innovation.I think thet the Open Source community has a lot to contribute here if we can find champions to actively promote the cause in Government. So even before we get to explaining the value of OpenNMS there is a lot of work to do to help Government wake up to understand how to organise themselves so as to exploit open source solutions effectively. We hope to be using the SOCTIM channel going forwards to encourage this debate.

 

About Shadowcat

Shadowcat Systems is an open source software developer and software consultancy provider based in the UK but accustomed to operating worldwide via electronic communications and distributed development environments. They offer proven expertise in development of networked systems and reliably automating manual processes from business workflow to systems and network management.

Shadowcat have the experience to deliver fully integrated open source solutions which are managed by OpenNMS.

Contact Shadowcat

128 The Barracks White Cross South Road Lancaster LA1 4XQ

+44 (0)1524 842155

http://shadow.cat

E-mail addresses info@shadowcat.co.uk

 

About OpenNMS UK

Entimoss Ltd (OpenNMS UK) is a full partner with OpenNMS Group Inc

They act as evangelists for the OpenNMS project and represent the community into technical communities such as the Telemanamgent Forum and the UK Network Operators Forum

They sell consulting, support and custom development services for users of the OpenNMS platform in the UK and Ireland.

 

Contact OpenNMS UK

Dr. Craig Gallen

entimOSS Ltd (OpenNMS UK)

6 Burnett Close, Bitterne Park, Southampton, SO181JD

+44 7789 938012 (m)

Skype: craig.gallen

email: craig.gallen@entimoss.com

cgallen@opennms.org

www.opennms.co.uk

Company registered in England and Wales No. 06402040

VAT Registration No. 140 6387 23

People sometimes ask us ‘Is OpenNMS ITIL compliant’, which is a strange question since it is your organisational processes which conform to ITIL, not your operational support systems. So it is more about how your use your systems than about the systems themselves. Recently we held OpenNMS workshops around the UK and Ireland and one of the presentations started to look at this question with a worked example of introducing a new service. To be honest, this example only scratched the surface but it does give you some pointers where to start. In this example I have taken a 3 tier WordPress cloud installation and started to think through how to manage it using OpenNMS . I hope to expand this example going forwards to begin pulling in other OpenNMS features to address other aspects of ITIL.

The presentation also illustrates how to monitor Varnish Cache using both SNMP and the XML data collector

The presentation video is available below and the example configurations and other material from the workshops are available in the workshop page in our public wiki here

The attached video is a repeat of a talk given by Dr Craig Gallen at the OUCE 2014 OpenNMS user conference suggesting that future Network Operations Centres (NOCs) will become more like Commodity Trading Desks.

In the future network, compute and storage infrastructure will all be virtualised and offered through ‘Cloud Services’. We expect that a market in trade-able cloud ICT resources will emerge and that the pressure to optimise the utilisation of large data centres will greatly increase. In this talk we will discuss the future trends and look at options for making OpenNMS a key component to help with ‘sweating the assets’ of a cloud service provider. We also present an early attempt to address this problem, the OpenNMS Pmatrix display, created in partnership with TMX Atrium, a leading low latency venue-neutral infrastructure provider for financial services participants in Europe and North America.

 

http://vimeo.com/93288426

Well the 2014 OpenNMS User Conference is now over (sigh). 

I think it went remarkably smoothly considering it was the first item we had hosted it in Southampton. Any feedback on the venue and choices of venue for next year would be appreciated.

Thanks  to everyone who attended and made such interesting contributions.  The program was recorded and the talks and slides will be available on the OpenNMS Foundation Europe site

 Thanks are especially due to Markus Neumann and Ronnie Tromner who put such sterling efforts into arranging the program, taking the bookings and handling the finances.

Here’s a photograph of some of us on the boat trip –  looking very relieved it was all over

Natilie Roseblade

We were very lucky to get Natilie Rosblade to sing for us during the Conference meal. It was lovely to have some female input into what turned out to be an  all male event ( can we get some ladies into the OpenNMS community?). I received nothing but compliments for her. So for those who missed it, here is some more…

http://www.youtube.com/watch?v=lbCuOf2Nq-U