KX and Databricks have partnered to develop time series analytics solutions for the capital markets sector to support many use cases including quant research and temporal trade-data analytics.
To date, data science and analytics programming languages such as SQL, Python or R for temporal or time-series analytics have been both cumbersome and time-intensive. Despite its popularity and powerful query language, SQL has limitations when interrogating time-series data about order (e.g., time-based joins) and prior states. Python and R, and even Spark, require pages of code to perform temporal analytics. These limitations are further compounded by the challenges of high-dimensional data associated with time-series analysis.
For hedge funds or institutional investors in particular, this collaboration combines the specialized series time-series data handling capabilities of KX with the comprehensive compute and machine learning frameworks available on Databricks. By focusing on time-series data, this partnership sets a new standard in quantitative and data science research, modeling, and trading analysis for the financial industry.
The combined strengths of KX and Databricks offer significant benefits, particularly in data management. Delta Lake provides a cost-effective, secure, and reliable means of managing time-series data supported by unified governance capabilities. The vast volume of data managed by clients leveraging Databricks is growing exponentially, and Databricks's partnerships with the major data providers will enable access to the data with ease, natively on the cloud, through mechanisms such as delta sharing.
It fosters collaboration across departments by enabling access to a unified data ecosystem, where a single copy of data is used by multiple teams. Users get to tap into scalable and price-performant compute on a single version of data, managed by Unity Catalog, the Databricks governance framework, for cross-domain applications and data pipelines, using a developer-friendly platform for instantaneous quantitative and trading workloads.
The merging of PyKX, KX's Python interoperability interface for the highly regarded kdb+ time-series database, with Databricks offers capital markets firms a powerful platform for performing sophisticated queries and analytics on extensive datasets stored in Delta Lake, eliminating the need for additional data storage solutions and simplifying the analytics workflow.
PyKX seamlessly installs via Python Package Index (PyPI) to be integrated into existing Databricks notebooks to handle increasingly larger datasets with speed and elegance and can natively execute on a single driver node within Databricks, delivering impressive performance with sizable in-memory datasets.
By focusing on time-series data analysis, the partnership enables a detailed exploration of financial data over time, unlocking crucial insights for strategic and in the moment decision-making.
Aside from time-series analysis, capital markets can leverage KX and Databricks for use cases that include enhanced strategy backtesting, market surveillance, counterparty risk analysis, and high-volume order book analysis. Asset management firms can perform predictive portfolio analysis and risk management strategies. Within banking, KX's application across fraud detection and prevention can help customers to perform fine-tooth-comb analysis across transactions, orders and market data to pinpoint nefarious trading behaviors indicative of fraud by utilizing the best-in-class temporal analytics of kdb+ alongside Databricks' machine learning frameworks to refine fraud detection models at low latency continuously. This integration equips banks to effectively combat fraud, ensuring operational security and protecting customer assets.
Beyond banking and capital markets, the insurance and payments sectors can also benefit from Delta Lake's transactional capabilities alongside KX's advanced time-series analytics to improve use cases such as pricing, claims and premium forecasting, and fraud detection and mitigation.
The KX and Databricks integration enhances Databricks' capabilities in time-series data analysis. It paves the way for further advancements in our mutual customers' machine learning (ML) interoperability, allowing them to advance existing investments seamlessly using KX on Databricks. Together, we are working towards integrating ML algorithms that can be applied to the vast datasets managed within the Databricks Lakehouse alongside KX and kdb+ to uncover more profound and more timely insights, predict trends, and improve mission-critical decision-making. This approach leverages the strength of kdb+ in handling large-scale time-series data and analytics, with the advanced data engineering and ML/AI capabilities of Databricks, providing a comprehensive analytics platform for the next generation of capital markets.
Furthermore, the integration offers seamless access to traditional KX solutions deployed on-premises or in the cloud. This ensures that firms can leverage the full spectrum of KX's analytical capabilities with the flexibility and scalability of Databricks' cloud-native platform for enhanced Data Science/ML and notebooking connectivity into a KX customer's estate. By enabling seamless access to these powerful tools, the partnership ensures that capital markets professionals have a robust set of solutions for data analytics, capable of meeting a wide range of analytical needs from historical data analysis up the stack to low-latency and real-time streaming insights.
This integration merges KX's specialized capabilities with Databricks' scalable analytics platform, enabling Capital Markets firms to navigate today's evolving data-driven landscape more effectively.
Connor Gervin, Lead Architect and Jack Kiernan, Head of Sales, Americas at KX will be sharing more about KX's collaboration with Databricks at this year's Databricks Financial Services Forum in New York on March 6. You can also watch the KX demo from the on-demand replay in April.
