Abdulla Al-Qawasmeh is an engineering leader who has managed artificial intelligence (AI) and big data teams. He has held engineering leadership positions at several companies in diverse industries including: Social Media, Financial Technology (FinTech), and Marketing Intelligence. He currently leads the Communities AI team at LinkedIn, which uses AI to help build communities centered around areas of interest on the LinkedIn platform. He holds a Ph.D. degree in Computer Engineering. His Ph.D. research focused on optimizing scheduling in large-scale data centers.
June 25, 2020 05:00 PM PT
The Communities AI team at LinkedIn generates follow recommendations from a large (10's of millions) set of entities to each of our 690+ million members. These recommendations are driven by ML models that rely on three sets of features (member, entity, and interaction features). In order to support a fast-growing user base, an expanding set of recommendable entities (members, companies, hashtags, groups, newsletters etc.) and more sophisticated modeling approaches, we have re-engineered the system to allow for efficient offline scoring in Spark. In particular, we have handled the 'explosive' growth of data by developing a 2D Hash-Partitioned Join algorithm that optimizes the join of hundreds of terabytes of features without requiring significant data shuffling. In addition to a 5X runtime performance gain, this opened the opportunity for training and scoring with a suite of non-linear models like XGBoost, which improved the global follow rate on the platform by 15% and downstream engagement on LinkedIn feed from followed entities by 10%.