An Experimentation Pipeline for Extracting Topics From Text Data Using PySpark
This post is part of a series of posts on topic modeling. Topic modeling is the process of extracting topics from a set of text documents. This is useful for understanding or summarizing large collections of text documents. A document can be a line of text, a paragraph or a chapter in a book. The...
Databricks Lecture Series at UC Berkeley School of Information
This is a collaborative post from Databricks and UC Berkeley. We thank Tia Foss, Director of Philanthropy, UC Berkeley School of Information, for her contributions. Databricks began in the computer labs of the University of California, Berkeley, where talented computer science graduate students, such as Matei Zaharia and Reynold Xin, realized that open-source, cloud-based scalable...
Augment Your SIEM for Cybersecurity at Cloud Scale
Over the last decade, security incident and event management tools (SIEMs) have become a standard in enterprise security operations. SIEMs have always had their detractors. But the explosion of cloud footprints is prompting the question, are SIEMs the right strategy in the cloud-scale world? Security leaders from HSBC don’t think so. In a recent talk,...
Getting Started With Ingestion into Delta Lake
Ingesting data can be hard and complex since you either need to use an always-running streaming platform like Kafka or you need to be able to keep track of which files haven’t been ingested yet. In this blog, we will discuss Auto Loader and COPY INTO, two methods of ingesting data into a Delta Lake...
The Delta Between ML Today and Efficient ML Tomorrow
Delta Lake and MLflow both come up frequently in conversation but often as two entirely separate products. This blog will focus on the synergies between Delta Lake and MLflow for machine learning use cases and explain how you can leverage Delta Lake to deliver strong ML results based on solid data foundations. If you are...
Monitoring ML Models With Model Assertions
This is a guest post from the Stanford University Computer Science Department. We thank Daniel Kang, Deepti Raghavan and Peter Bailis of Stanford University for their contributions. Machine learning (ML) models are increasingly used in a wide range of business applications. Organizations deploy hundreds of ML models to predict customer churn, optimal pricing, fraud...
Improving Patient Insights With Textual ETL in the Lakehouse Paradigm
This is a collaborative post from Databricks and Forest Rim Technology. We thank Bill Inmon, Founder and CEO, and Mary Levins, Chief Data Officer, of Forest Rim for their contributions. The amount of healthcare data generated today is unprecedented and rapidly expanding with the growth in digital patient care. Yet much of the data...
Top Considerations When Migrating Off of Hadoop
Apache Hadoop was created more than 15 years ago as an open source, distributed storage and compute platform designed for large data sets and large-scale batch processing. Early on, it was cheaper than traditional data storage solutions. At the time, businesses didn't need to run it on particular hardware. The Hadoop ecosystem also consists of...
The Three Things CXOs Prioritize in Their Data and AI Strategy
Leveraging data (internal and external) and customer analytics to innovate and create competitive advantages is more powerful than it has ever been. This popular practice is fueled by the growing volume of operational and customer data and technological advancements that make extracting value from data even faster and more accessible. Driving more value from all...
Unlocking The Power of Health Data With a Modern Data Lakehouse
A single patient produces approximately 80 megabytes of medical data every year. Multiply that across thousands of patients over their lifetime, and you’re looking at petabytes of patient data that contains valuable insights. Unlocking these insights can help streamline clinical operations, accelerate drug R&D and improve patient health outcomes. But first, the data needs to...
