Advancing sustainable investment in emerging markets
Sentences analyzed per
minute — a 950x increase
ESG risk terms
rapidly identified
As emerging markets seek to attract more capital to spur economic development, investors are considering ESG issues such as impacts on local communities, value of goods generated, and greenhouse gas emissions when selecting investments. The International Finance Corporation (IFC) advances economic development and improves the lives of people by encouraging the growth of the private sector in developing countries. Using MALENA — IFC’s AI-powered ESG analytics platform — users can extract meaningful insights from unstructured ESG data at scale, enabling rapid analysis, increasing productivity, and building confidence.
IFC has successfully scaled the MALENA platform through the use of Databricks Data Intelligence Platform and leveraged the distinct advantages offered by the lakehouse architecture namely, optimal connectivity with Azure Data Lake coupled with GPU computing, features supporting the development of machine learning solutions, and robust and consistent infrastructure for data engineering and machine learning. Shifting to Databricks will also allow IFC to deliver MALENA to external users as the World Bank Group’s first AI as a Service. By using MALENA, emerging market investors will be able to quickly extract insights from unstructured data to identify better quality investments.
IFC’s Machine Learning ESG Analyst (MALENA) is an Artificial Intelligence (AI) powered platform that extracts meaningful insights from unstructured ESG data at scale, enabling rapid analysis, increasing productivity, and building investor confidence.
Harnessing AI for Sustainable Investing in Emerging Markets: IFC's MALENA
A member of the World Bank Group, IFC is the largest global development institution focused on making investments in the private sector in emerging markets. IFC understands the importance of making investment decisions guided by responsible and sustainable principles. “When making investments, we look carefully at the sustainability impact of a client’s business and consider a multitude of angles such as a company’s track record for worker safety, methods used for pollution prevention, or approaches to engaging stakeholders,” explained Mary Porter Peschka, Director Sustainability and Gender Solutions at IFC. “It is important for IFC to use all possible solutions to support our clients and emerging markets, including innovation and data science. Collaboration with Databricks is essential to deliver applications on time and of high quality.”
To achieve this mission, IFC built MALENA or Machine Learning ESG Analyst. MALENA is an AI-powered platform that supports the review and analysis of massive amounts of text to identify ESG opportunities and risks. MALENA is designed for IFC as well as for other users such as institutional investors, asset owners, asset managers, multilateral development banks, commercial banks, private equity funds, regulators, capital markets authorities, and data providers seeking to increase ESG investments in emerging markets while tackling constraints to ESG data and analytical capacity. At the press of a button, users can unlock curated information from many lengthy documents, PDFs, and tables running into thousands of pages, disregard the non-essential information, and focus on what matters.
Leveraging the Uniqueness of Databricks Data Intelligence Platform at IFC
As the demand for ESG insights and the volume of unstructured textual data continues to increase, IFC recognized the need for a data science platform to support MALENA’s NLP and large language model training needs. Previously reliant on manual processes and limited data processing tools, the operationalization of ML models to analyze roughly 21 million pages of text took months to complete. Before the Databricks Data Intelligence Platform, “training and fine-tuning large language models was limited and resulted in frequent failures. And limited cluster choices impacted our ability to optimize model performance,” explained IFC CIO Laura McCanlies.
Additionally, user experience was not ideal in this setup with limited flexibility to share code across teams. This prevented the collaboration IFC needed to operate efficiently. “We knew we needed a robust platform to get the job done and after running a proof of value, Databricks Data Intelligence Platform was an obvious choice,” said Laura.
Building a Smarter Approach to ESG Analysis with Lakehouse Architecture
By leaning on the power of Databricks Data Intelligence Platform, IFC experienced the advantages of GPU computing that scaled on-demand to support analytics and ML workloads. With Azure Data Lake and Workflows, they were able to create and orchestrate data pipelines for ingestion, transformation, and scheduled pipeline execution. MLflow and Model Registry provide the tooling necessary to track model performance, versioning, and artifacts throughout the entire ML lifecycle. The lakehouse has enabled IFC to access faster data processing and run large-scale inferences and state-of-the-art models that deliver key use cases. As a result, IFC is able to easily integrate and complement its legacy unstructured data with external and alternative ESG data and gain more comprehensive coverage in identifying ESG risk factors.
For example, IFC has trained sentiment analysis models to extract insights from across hundreds of thousands of documents and millions of lines of text within impact assessments, news articles, and sustainability reports — helping users make ESG-integrated decisions. Additionally, IFC has improved the MALENA platform user experience by enabling users to view ESG sentiment predictions directly in the source PDFs and Word documents. To provide comprehensive levels of visibility into investment opportunities, named entity recognition is used to integrate external data sources and connect news stories to client profiles. As a result, IFC has developed 10,000 company profiles and expanded insights for more than 180 markets. And to ensure the performance of all their models over time, IFC’s MLOps leverages a Transparency dashboard in PowerBI to monitor model prediction performance, confidence, drift, explainability, robustness, and fairness.
Looking ahead, IFC plans to migrate to the Unity Catalog to simplify data access management, improve data observability, and enhance data sharing within IFC and with other institutions. With the launch of Databricks Model Serving, another priority is deploying ML models as a REST API, delivering a game-changing real-time and low-latency inference experience for external users of the MALENA document upload feature.
Investing in AI With an Eye Toward Creating ESG Capacity for Others
With Databricks Data Intelligence Platform as the backbone for its MALENA platform, IFC now has the means to solve its scalability challenges and achieve its goals to increase capacity to analyze ESG issues in emerging markets through deploying AI. For starters, in-house document analysis times have been reduced from weeks to days for ESG due diligence. MALENA can analyze 19,000 sentences per minute; compared with the average human reader who can only read 15 to 20 sentences per minute — that is a 950x increase.
With access to such ESG analytical capabilities, IFC is expanding its support for emerging markets by strengthening the capacity of other investors to also make ESG-driven decisions for investments in green and climate assets, gender-smart products, and sustainability bond issuances. IFC will do this by providing external users access to the MALENA AI to extract meaningful insights from unstructured ESG data at scale, enabling rapid analysis, increasing productivity, and building confidence.
“The operational efficiencies that we’ve gained using the Databricks Data Intelligence Platform have paid off in spades,” said Laura. “We can now lean on the power of GPU computing and AI to support our global ESG initiatives – leading to better quality investments and a more sustainable future.”
