FINRA

To protect investors, FINRA is involved in identifying fraud schemes in the market. It does so by surveilling 99 percent of the equity markets and approximately 70 percent of the options markets. To do this, FINRA captures all transaction data—more than 100 billion trading events per day—from the various securities markets. FINRA then uses machine-learning algorithms to identify fraudulent patterns of behavior for investigation. Unfortunately, their legacy architect created a number of challenges that prevented them from effectively and efficiently monitoring the security markets. Challenges included:

• Data was stored in disparate on-premise systems which were highly complex and costly to build and scale leading to brittle data pipelines
• Disjointed development and production systems required data engineers to convert Python-based machine learning models into complex SQL statements that often times totaled 60-70 pages of code
• Complex model development process made it challenging to debug and iterate on models and limited reuse of code across teams
• Long development cycles due to segmented data scientist and engineering teams