Ruihui Zhao - Databricks

Ruihui Zhao

Research Scientist, Tencent

Ruihui Zhao is a research scientist at Tencent, applying NLP and deep learning techniques to solve different types of business problems. He served as a reviewer of TPDS, ACL etc., and published several papers at WCSP, LISS, and IJCAI. Ruihui received his B.S. and M.S. degrees in computer engineering at UESTC and Waseda University, respectively.


Explainable Anomaly Detection Using Apache SparkSummit 2020

Security operation center (SOC) enables enterprise to monitor and analyze organizational-wise security postures. SOC platform monitors invoking behaviors of software in devices and generates tens of millions of behavior event sequence data daily. In this work, we propose an explainable anomaly detection method using Spark AI, which automatically discovers new threat patterns from event sequence data and provides reason code for security operations. The results are further enhanced through interactive visualization. Since we need to handle tens of millions of records per day, we rely on spark for large scale data analytics and modeling. We leveraged an unsupervised anomaly detection algorithm based on Variational AutoEncoders (VAE) in this work. The model learns latent representations for all sequences of events and detects anomalies that deviate from the overall distribution using One-Class SVM. We also employ a visualization system to facilitate interpretations of anomalies and attributes for security operation. Finally, we quantitatively evaluate the performance of our anomaly detection model and demonstrate the effectiveness of our system through report and feedback collected from SOC platform.