Building Industry IoT and M2M Solutions With Databricks for Communications

The communications industry is experiencing immense change due to rapid technological advancements and evolving market trends. Communications service providers (CSP) build various solutions to manage their networks for monitoring and optimisation and personalised experiences for their customers. With the broader implementation of 5G networks by CSPs and significant investments in IoT (Internet of Things), M2M (Machine to Machine) solutions across industries such as automotive, manufacturing, retail, healthcare, and logistics, CSPs are uniquely positioned to increase their revenue by monetising their networks with additional solutions and services. This blog focuses on building IoT and M2M solutions with Databricks. We will see how challenging it can be for CSPs to create DIY(do-it-yourself) Data and AI solutions for industry IoT and M2M use cases and how Databricks Intelligence Platform for Communications can help CSPs build these industry solutions faster with lower TCO and higher ROI.

Challenges in Building Data, AI Solutions for IoT and M2M Use Cases

Building Data and AI solutions present several challenges:

• Data Management: IoT devices and M2M communication generate significant data at the edge, which is expected to reach up to 79.4 zettabytes (ZBs) by 2025. Data processing solutions of the past cannot keep up with data at this scale, and modern solutions are necessary. Data management typically involves collecting, storing, and utilising data efficiently and securely from diverse data sources. This can be challenging, but it's essential for effectively training AI models, and this requires a strategic approach that includes robust data governance frameworks.
• Data Quality and Integration: AI models rely on high-quality datasets. Issues like inaccuracies can lead to poor AI performance. Integrating data from multiple IoT devices adds complexity. Organisations must prioritise data validation, cleaning, and integration processes to enhance data quality.
• Data Privacy and Security: Using and processing personal and confidential information, such as in retail solutions with IoT-based cameras, sensors, and other devices, raises ethical and legal issues and requires strict measures to protect data. Compliance with regulations like GDPR, HIPAA, etc., and using advanced security technologies is crucial and a top priority.
• Scalability: Systems must handle increasing data volumes from IoT devices and long-term storage requirements (up to 10 years for some industries) due to regulations without compromising performance. This requires scalable storage, distributed computing, and efficient algorithms. Organisations must design cloud-based modular systems architectures with scalability in mind.
• Regulatory, Ethical, and Safety Concerns: As AI systems take on more decision-making roles, regulations such as the EU AI Act emphasise the need to ensure fairness, transparency, and accountability for such systems. Addressing these issues to comply with evolving regulations poses challenges for organisations.

In summary, while AI and data solutions offer significant potential benefits, they also come with substantial challenges related to data quality, integration, regulatory compliance and infrastructure.

Databricks and Communications Industry - A Better Together Partnership

Databricks offers several unique advantages to Communications service providers (CSP) that they can leverage to build Data and AI solutions for industries:

Data Intelligence Platform for Communications

Databricks, with its Data Intelligence Platform for Communications, addresses these challenges by collaborating closely with CSPs. This collaboration enhances data-driven decision-making and leverages the power of AI and machine learning to help CSPs optimise network performance and customer interactions and build IoT and M2M solutions for industries. Databricks provides a comprehensive platform for building Data and AI solutions with a unified governance solution with Unity Catalog, an open protocol for sharing data, and AI assets with Delta Sharing, a platform to build and deploy production-quality ML and GenAI applications with Mosaic AI, intelligent analytics with AI/BI, intelligent data warehouse with Databricks SQL and real-time analytics applications with Data Streaming. With all the necessary features and services pre-integrated, CSPs can focus on building solutions for industry use cases rather than building and integrating the platform components and continue investing in keeping the platform updated with future technology advancements.

Hybrid Architecture With Cloud and Edge Deployments

Edge computing plays a critical role in building IoT and M2M solutions. Most use cases involve data collection from IoT edge devices and running machine learning (ML) models at the edge on a 5G MEC (Multi-Access Edge Computing) environment that provides storage and computing at the edge. One challenge for Data Scientists and ML engineers working on training and deploying these ML models is the variety of 5G MEC edge devices with different operating systems and machine learning runtimes they support. This becomes a challenge as the number of edge devices and ML models scale up.

With the MLOps-driven approach using MLflow on Databricks, data scientists and engineers can simplify the complexity of managing these ML runtimes and deployment environments using MLflow flavours, a key feature of MLflow. They provide a consistent API for different machine learning libraries and can be used to save models in multiple formats, or "flavours". Databricks ML Runtime provides data scientists and practitioners with scalable clusters, including popular frameworks, built-in AutoML, and performance optimisations for managing ML runtimes. The ML Runtime offers one-click access to reliable and performant distribution of the most popular ML frameworks and custom ML environments via pre-built containers. All this helps data scientists and ML engineers focus their efforts on use cases and business requirements that require them to build custom ML models. At the same time, MLflow abstracts the complexities of deploying them on various edge devices with different runtimes.

Consider a case where an Electric Vehicle (EV) manufacturer or charging solution provider relies on CSPs 5G MEC to run ML inference for anomaly detection for EV charging use cases. With isolated and governed workspaces for Telcos and EV manufacturers, CSPs can collect charging data at the edge and process and transform it in their workspaces. Databricks has a rich ecosystem of partners and real-time IoT data ingestion streaming capabilities already built into the platform. Workflow capabilities are available as a managed orchestration service for ETL processing and transformation. After data transformation and processing, CSPs can use Delta Sharing to share processed data with EV manufacturers' workspaces. They then use this data to build and train their own ML models for inference using the MLOps driven approach using MLflow on Databricks. After model training, EV manufacturers can again use Delta Sharing to share trained models with CSPs who can manage the deployment of ML models on 5G MEC computing environments.

Databricks provides a ready-made Solution Accelerator for Bringing Scalable AI to the Edge, which CSPs can use and extend to customers seeking similar Edge AI inference solutions.

Multi-Cloud Architecture and Open Scalable Data Sharing

Another pressing challenge for CSPs and their customers is the need to build cloud-agnostic solutions as part of their multicloud strategy either due to regulations like the Digital Operational Resilience Act (DORA) in the EU for the financial industry or more as a strategic choice either to avoid vendor lock-in or for cost efficiency reasons. Hence, this results in a situation where the CSPs have a specific cloud provider and must collaborate with customers who prefer a different one. Cross-cloud communication then brings additional challenges concerning complexity in networking and connectivity, security and compliance, additional management and monitoring overhead, additional costs, integration and interoperability challenges. Focussing on solving these issues diverts the focus and investments of CSPs away from building IoT and M2M solutions.

Databricks is designed to be a cloud agnostic Data and AI platform, meaning it can run workloads similarly across any cloud platform, whether AWS, Azure, or GCP. This flexibility allows CSPs to build data and AI solutions once and provide them to their customers, who would run them in their own Databricks environment on any of the supported cloud platforms. Additionally, Databricks enables open data sharing for data, analytics, and AI assets using Delta Sharing, an open protocol developed by Databricks and the Linux Foundation. It allows secure, real-time exchange of large datasets across various computing platforms, including cloud and on-premises environments.

Read how Delta Sharing enables customers like Deutsche Börse, Shell and Nasdaq to promote interoperability and collaboration across cloud platforms.

Solution Accelerators for Industries

Databricks has introduced a suite of Solution Accelerators as part of its Data Intelligence Platform for Communications, designed to expedite the deployment of data analytics and AI solutions. These accelerators are pre-built guides, including fully functional notebooks and best practices, to address daily and high-impact use cases across industries. They are designed to save hours of discovery, design, development, and testing, enabling organisations to move from idea to proof of concept in a significantly reduced timeframe. Let's take a look at some of them:

• On-Shelf Availability: Out-of-stock (OOS) is one of the biggest problems in retail and supply chain. This Solution Accelerator shows how OOS can be solved with real-time data and analytics, using data collected from IoT devices and RFID (Radio Frequency Identification) sensors and processed in real-time with streaming ingestion and processing capabilities. Using this accelerator, CSPs can build solutions for their retail and supply chain customers to improve on-shelf availability in real-time and increase retail sales.
• Grid-Edge Analytics: For Energy providers, it is crucial to leverage data from the edge of the grid to make informed decisions to optimise energy grid performance and prevent outages. The accelerator helps CSPs build solutions for energy providers to unify data from IoT devices like smart meters and sensors, analyse it for deeper insights into grid behaviour, and train a convolutional neural network-based fault detection model to identify anomalies. This approach aims to manage energy demands, enhance grid performance, and reduce greenhouse gas emissions.

Here is a complete list of solutions for other industries, such as retail and consumer goods, healthcare and life sciences, media, and entertainment, that CSPs can use and extend to build custom solutions for their customers.

Conclusion

As more businesses adopt IoT and M2M solutions to increase business efficiency and optimise their operations, CSPs can monetise their investments in 5G networks and provide value-added services to their customers. Also, 5G technology offers the possibility of building real time use cases, with their high bandwidth capacities that can deliver up to 20 Gigabits per Second (Gbps) of data with a low latency rate of up to 1 millisecond. We saw how challenging it can be for CSPs to build these platforms themselves and how Databricks provides a ready Data and AI platform that CSPs can use off-the-shelf and start using immediately to build IoT and M2M solutions. Also, being a managed platform that CSPs can deploy on their cloud of choice, they get access to all great features and regular upgrades that can help them build Data and AI solutions using the latest technologies.

Explore the Data Intelligence Platform for Communications, which provides all the capabilities discussed above, and test-drive the Databricks Platform free for 14 days on your choice of cloud.