Augment Your SIEM for Cybersecurity at Cloud Scale
Over the last decade, security incident and event management tools (SIEMs) have become a standard in enterprise security operations. SIEMs have always had their detractors. But the explosion of cloud footprints is prompting the question, are SIEMs the right strategy in the cloud-scale world? Security leaders from HSBC don’t think so. In a recent talk,...
Getting Started With Ingestion into Delta Lake
Ingesting data can be hard and complex since you either need to use an always-running streaming platform like Kafka or you need to be able to keep track of which files haven’t been ingested yet. In this blog, we will discuss Auto Loader and COPY INTO, two methods of ingesting data into a Delta Lake...
The Delta Between ML Today and Efficient ML Tomorrow
Delta Lake and MLflow both come up frequently in conversation but often as two entirely separate products. This blog will focus on the synergies between Delta Lake and MLflow for machine learning use cases and explain how you can leverage Delta Lake to deliver strong ML results based on solid data foundations. If you are...
Monitoring ML Models With Model Assertions
This is a collaborative post from Databricks and the Stanford University Computer Science Department. We thank Daniel Kang, Deepti Raghavan and Peter Bailis of Stanford University for their contributions. Machine learning (ML) models are increasingly used in a wide range of business applications. Organizations deploy hundreds of ML models to predict customer churn, optimal...
Improving Patient Insights With Textual ETL in the Lakehouse Paradigm
This is a collaborative post from Databricks and Forest Rim Technology. We thank Bill Inmon, Founder and CEO, and Mary Levins, Chief Data Officer, of Forest Rim for their contributions. The amount of healthcare data generated today is unprecedented and rapidly expanding with the growth in digital patient care. Yet much of the data...
Top Considerations When Migrating Off of Hadoop
Apache Hadoop was created more than 15 years ago as an open source, distributed storage and compute platform designed for large data sets and large-scale batch processing. Early on, it was cheaper than traditional data storage solutions. At the time, businesses didn't need to run it on particular hardware. The Hadoop ecosystem also consists of...
The Three Things CXOs Prioritize in Their Data and AI Strategy
Leveraging data (internal and external) and customer analytics to innovate and create competitive advantages is more powerful than it has ever been. This popular practice is fueled by the growing volume of operational and customer data and technological advancements that make extracting value from data even faster and more accessible. Driving more value from all...
Unlocking The Power of Health Data With a Modern Data Lakehouse
A single patient produces approximately 80 megabytes of medical data every year. Multiply that across thousands of patients over their lifetime, and you’re looking at petabytes of patient data that contains valuable insights. Unlocking these insights can help streamline clinical operations, accelerate drug R&D and improve patient health outcomes. But first, the data needs to...
AML Solutions at Scale Using Databricks Lakehouse Platform
Anti-Money Laundering (AML) compliance has been undoubtedly one of the top agenda items for regulators providing oversight of financial institutions across the globe. As AML evolved and became more sophisticated over the decades, so have the regulatory requirements designed to counter modern money laundering and terrorist financing schemes. The Bank Secrecy Act of 1970 provided...
Feature Engineering at Scale
Feature engineering is one of the most important and time-consuming steps of the machine learning process. Data scientists and analysts often find themselves spending a lot of time experimenting with different combinations of features to improve their models and to generate BI reports that drive business insights. The larger, more complex datasets with which data...
