Application Spotlight: Technicolor Virdata Internet of Things platform

About Virdata

Virdata is Technicolor’s cloud-native Internet of Things platform offering real-time monitoring, configuration and management of the unprecedented number of connected devices and applications. Combining its highly-scalable data ingestion and messaging capabilities with real-time and historical analytics, Virdata brings value across multiple data-driven markets.

The Virdata platform was launched at CES Las Vegas in January, 2014.
The Virdata cloud-based platform architecture integrates state-of-the-art open source software components into a homogeneous, high-availability data-processing environment.

Virdata and Apache Spark

The Virdata solution architecture comprises 3 areas: Messaging, Data Processing and Applications - all accessed through APIs. Its publish/subscribe based messaging infrastructure contains a high-throughput distributed message broker and distributed complex event processing and bidirectional message routing components.

Completing Virdata's “full stack” Internet of Things (IoT) platform, the solution provides extensive Apache Spark-driven in-memory and post-processing capability in order to transform, store and analyze the huge stream of messages generated by the world of IoT.

Spark was integrated in the Virdata architecture in early 2013 as the data processing framework to analyze incoming IoT messages published by the millions of devices monitored by the Virdata cloud platform. Spark is used in Virdata for batch processing and real-time processing of device data in order to compute time-series, pre-calculated complex data visualizations and custom monitoring reports.

Virdata’s data processing implementation incorporates a Lambda architecture (http://manning.com/marz/) well-suited to a combination of Spark and Spark Streaming.

Virdata migrated its data processing approach from the initial stream-oriented framework where every message was processed independently to adopting Spark Streaming in order to optimize processing and message storage in a distributed manner. The Spark Streaming micro-batching approach specifically enabled eliminating any occurrences of ‘impedance mismatch’ with the storage of data in the Virdata databases.

The growing adoption of Spark by the open-source community plays an important role. As just one example, the recently released Spark-Cassandra Driver has allowed Virdata to replace custom code with a specialized component, delivering improved performance characteristics.

How Virdata benefits from Spark

Virdata benefits from Spark in 3 ways:

• Spark offers Virdata a single framework for both batch and real-time processing.
• Spark offers the programming languages favored by its own developers and the wider data science community.
• Spark supports Virdata's native cloud dev-op and configuration environment.

Virdata and Spark Going Forward

Virdata is especially excited about the growing richness of libraries in the Spark ecosystem and is already considering the integration of additional functionality such as SparkSQL, the recently announced Spark-ElasticSearch integration and the Spark Machine Learning library (MLlib) and looks forward to the announcement of many others .

To learn more about Technicolor Virdata, please visit our website at www.technicolor.com and feel free to send us an email at [email protected] for questions and demonstrations.