Skip to main content
Page 1

Cost-effective, incremental ETL with serverless compute for Delta Live Tables pipelines

August 27, 2024 by Paul Lappas, Michael Armbrust, Manish Dalwadi, Andreas Neumann, Xi Liang, Kiavash Kianfar and Karl Wang in Platform Blog
We recently announced the general availability of serverless compute for Notebooks, Workflows, and Delta Live Tables (DLT) pipelines. Today, we'd like to explain...

Build Reliable and Cost Effective Streaming Data Pipelines With Delta Live Tables’ Enhanced Autoscaling

December 7, 2022 by Paul Lappas, Li Zhang, Alex Ott, Kiavash Kianfar, Yuhong Chen and Prashanth Babu Velanati Venkata in Platform Blog
This year we announced the general availability of Delta Live Tables (DLT) , the first ETL framework to use a simple, declarative approach...

Integrating large-scale Genomic Variation and Annotation Data with Glow

October 9, 2020 by Kiavash Kianfar in Engineering Blog
Genomic annotations augment variant data by providing context for each change in the genome. For example, annotations help answer questions like does this...

Introducing GlowGR: An industrial-scale, ultra-fast and sensitive method for genetic association studies

June 25, 2020 by Leland Barnard, Henry Davidge, Karen Feng, Joelle Mbatchou, Boris Boutkov, Kiavash Kianfar, Dr. Lukas Habegger, Jonathan Marchini, Dr. Jeff Reid, Evan Maxwell and Frank Austin Nothaft in Product
Today, we announce that we are making a new whole genome regression method available to the open source bioinformatics community as part of...

Glow 0.3.0 Introduces New Large-Scale Genomic Analysis Features

April 23, 2020 by Kiavash Kianfar in Engineering Blog
In October of last year, Databricks and the Regeneron Genetics Center ® partnered together to introduce Project Glow , an open-source analysis tool...

Streamlining Variant Normalization on Large Genomic Datasets with Glow

December 4, 2019 by Kiavash Kianfar in Engineering Blog
Cross posted from the Glow blog . Many research and drug development projects in the genomics world involve large genomic variant data sets...

Introducing Glow: An Open-Source Toolkit for Large-Scale Genomic Analysis

October 17, 2019 by Frank Austin Nothaft, Karen Feng, Henry Davidge, Ion Stoica, Dr. Jeff Reid, Dr. Lukas Habegger, Evan Maxwell, Leland Barnard and Kiavash Kianfar in Announcements
The key to solving some of today’s most challenging medical problems lies in the analysis of genomics data. Understanding the impact of the...