Engineering Blog › Streaming

The proliferation of subscription models has increased across industries: from direct-to-consumer brands for shaving supplies and prepared meals to streaming media services, at-home fitness, auto insurance and even automobiles themselves. Consumers are flocking to these new offerings while moving away from long-term contracts, which for subscription-based businesses means they have to prove their value to...

This is a guest community post from Genmao Yu, a software engineer at Alibaba. Structured Streaming was initially introduced in Apache Spark 2.0. It has proven to be the best platform for building distributed stream processing applications. The unification of SQL/Dataset/DataFrame APIs and Spark’s built-in functions makes it easy for developers to achieve their complex...

Download the notebooks to demo the solution covered below The subscription model is experiencing a renaissance.  Gone are the days of the penny music CD clubs, replaced by an ever-increasing assortment of digital streaming services delivering music, videos and more directly to consumers’ devices in exchange for a modest recurring fee. Today, 70% of US...

Cloud computing has fundamentally changed how companies operate - users are no longer subject to the restrictions of on-premises hardware deployments such as physical limits of resources and onerous environment upgrade processes. With the convenience and flexibility of cloud services comes challenges on how to properly monitor how your users utilize these conveniently available resources....

Contents The Importance of Quality to Streaming Video Services Databricks QoS Solution Overview Video QoS Solution Architecture Making Your Data Ready for Analytics Creating the Dashboard / Virtual Network Operations Center Creating (Near) Real Time Alerts Next steps: Machine learning Getting Started with the Databricks Streaming Video Solution

Initially published April 14th, 2020; updated April 21st, 2020 With the massive disruption of the current COVID-19 pandemic, many data engineers and data scientists are asking themselves “How can the data community help?" The data community is already doing some amazing work in a short amount of time including (but certainly not limited to) one...

Try this notebook to reproduce the steps outlined below In the era of accelerating everything, streaming data is no longer an outlier- instead, it is becoming the norm. We often no longer hear customers ask, "can I stream this data?" so much as "how fast can I stream this data?", and the pervasiveness of technologies...

We are excited to announce the release of Delta Lake 0.5.0, which introduces Presto/Athena support and improved concurrency. The key features in this release are: Support for other processing engines using manifest files (#76) - You can now query Delta tables from Presto and Amazon Athena using manifest files, which you can generate using Scala,...