AI Trends that Paved the Way in 2018
With 2018 behind us, it’s been amazing to see AI projects gain steam and make significant impact across industries. In fact, a recent survey by CIO.com cites that 90% of enterprises are actively investing in AI. What has fueled this innovation is the massive influx of organizations tapping into the potential of their data and...
Apparate: Managing Libraries in Databricks with CI/CD
This is a guest blog from Hanna Torrence, Data Scientist at ShopRunner. Introduction As leveraging data becomes a more vital component of organizations' tech stacks, it becomes increasingly important for data teams to make use of software engineering best-practices. The Databricks platform provides excellent tools for exploratory Apache Spark workflows in notebooks as well as...
Open Sourcing Databricks Integration Tools at Edmunds
This is a guest post from Shaun Elliott, Data Engineering Tech Lead and Sam Shuster, Staff Engineer at Edmunds. What is Databricks and How is it Useful for Edmunds? Databricks is a cloud-based, fully managed, big data and analytics processing platform that leverages Apache SparkTM and the JVM. The big selling point of the Databricks Unified...
Introducing Flint: A time-series library for Apache Spark
This is a joint guest community blog by Li Jin at Two Sigma and Kevin Rasmussen at Databricks; they share how to use Flint with Apache Spark. Introduction The volume of data that data scientists face these days increases relentlessly, and we now find that a traditional, single-machine solution is no longer adequate to the demands...
Announcing Databricks Runtime 4.2!
We’re excited to announce Databricks Runtime 4.2, powered by Apache Spark™. Version 4.2 includes updated Spark internals, new features, and major performance upgrades to Databricks Delta, as well as general quality improvements to the platform. We are moving quickly toward the Databricks Delta general availability (GA) release and we recommend you upgrade to Databricks Runtime...
Viacom’s Journey to Improving Viewer Experiences with Real-time Analytics at Scale
With over 4 billion subscribers, Viacom is focused on delivering amazing viewing experiences to their global audiences. Core to this strategy is ensuring petabytes of streaming content is delivered flawlessly through web, mobile and streaming applications. This is critically important during popular live events like the MTV Video Music Awards. Streaming this much video can...
Overstock Marketing + Databricks = Data Science at Scale
This is a guest post from Chris Robison, Head of Marketing Data Science at Overstock.com. At Overstock.com we’ve never had a problem with a lack of data. At 19 years old, we have one of the most expansive user datasets in all of e-commerce. As Lead Data Scientist in Marketing, I can look back through...
Improving Threat Detection in a Big Data World
High-profile cybersecurity breaches dominated headlines in 2017. In the first half of the year, over 1.9B records were stolen. That’s more than 7,000 records breached every minute. And the fallout from a single event can be staggering. Customer attrition, negative PR and regulatory fines amount to millions in financial losses. In fact, according to recent...
Using Databricks to Democratize Big Data and Machine Learning at McGraw-Hill Education
This is a guest post from Matt Hogan, Sr. Director of Engineering, Analytics and Reporting at McGraw-Hill Education. McGraw-Hill Education is a 129-year-old company that was reborn with the mission of accelerating learning through intuitive and engaging experiences – grounded in research. When we began our journey, our data was locked in silos managed within...
On-Demand Webinar and FAQ: Accelerate Data Science with Better Data Engineering on Databricks
On July 13th, we hosted a live webinar — Accelerate Data Science with Better Data Engineering on Databricks. This webinar focused on the use of PySpark in transforming petabytes of data for ad-hoc analysis and generating downstream queries. In particular, we covered: Transforming TBs of data with RDDs and PySpark responsibly Using the JDBC connector...
