When Steve Rosenbush of the Wall Street Journal, commented in newspaper’s CIO Journal section, reported on June 15, that, “IBM Embraces Spark at Big Data’s Real-Time Frontier”; it signaled more than a new product services offering. It was a very big deal. IoT has been described in glowing terms; a panacea, the “Next Internet”, a solution to all our problems. Well. Maybe. But that dream has had little chance of becoming a meaningful economic reality until this announcement. This is a big deal.
IoT platforms and applications that properly scale, collect massive amounts of data. However, most IoT data remains at rest, (typically stored in Hadoop clusters) and therefore underutilized in after-the fact analysis. Spark, (or its alternatives), allow IoT data to be analyzed and acted upon concurrently. With massive amounts of data addressable in real-time, the use of Spark has the power to re-prioritize IoT investments with higher economic or social impact than ad serving, or social media monitoring.
A couple of examples that come to mind are:
- Real-time localized cyber security threat analysis
- Stress and structural failure warnings for physical infrastructure such as bridges, or dams
- Precise measurement of critical heat risks in data centers
- Toxin alerts in mines
- Continuous glucose monitoring of diabetic patients
- In motion recording of national entry boarders for risks of contagious diseases
- Instant ground war battle logistics.
Aside from the obvious immediate opportunities to convert and support meaningful services with real-time monitoring and analysis, streaming analytics provides increasing returns on investment the more it is used. With each data cycle, a properly designed IoT system that applies Spark can learn, adjust and become smarter at delivering even richer services and knowledge from the data gathered and acted upon in real-time. It is perhaps the first opportunity to remove the bottleneck of human interaction and data latency that have been restricting even greater IT scale at an affordable cost.
A major commercial endorsement of IoT streaming analytics, will result in the promotion of IoT to the Enterprise IT. In turn, massive IoT systems with higher returns in value are likely to be developed. Any enterprise considering IoT should take note of the reality, beyond the dream of a bold, smart future, enabled by Spark and streaming analytics.
All opinions & expressions are solely those of the author and not those of any other individual, institution or business.