Assembling Toy Brick Sets with Gurobi & Databricks: A Gentle Introduction to Optimization

As more and more organizations embrace analytics, a wider range of problems are being brought forward to be solved. While data science teams are often well-versed in traditional techniques like statistical analysis and machine learning, as well as emerging technologies such as AI, there still exists a class of problems that is more easily addressed using mathematical optimization.

Business functions are often tasked with making decisions that maximize the benefits of a process while managing multiple, sometimes conflicting, constraints. Unlike classical machine learning that predicts a future outcome based on current state variables, optimization helps the decision-makers to identify the set of actions required to best achieve a particular outcome. The solutions to these problems are rarely straightforward and require the examination of numerous, interacting components to identify the best solution. Some frequently encountered challenges of this type include:

• Product Assortment - finding the right mix of products to satisfy customer needs and maximize profits while dealing with limited shelf space
• Inventory - managing stock levels to minimize capital locked up in inventory while also being able to satisfy customer demand
• Pricing & Promotions - identifying the optimal base price and promotional discounts that maximize profits given the complexities of consumer demand and potential competitor responses
• Layout - identifying the ideal layout of goods on a shelf that maximize the revenue potential of a unit of space while dealing with variable product sizing and the need to provide consumers access to a range of product options
• Advertising - finding the right mix of advertising vehicles and channels, all of which differ in terms of their reach and cost, to maximize consumer response while minimizing investment
• Production Scheduling - allocating finite labor and material resources against a given production capacity to support the efficient and timely production of goods to meet demand
• Equipment Utilization - minimizing the downtime caused by equipment failure or inefficiencies through scheduled maintenance
• Logistics - identifying the appropriate bundling of items and routing of vehicles to meet delivery targets while working within driver and vehicle capacity constraints
• Supply Chain - balancing the delivery and storage of goods between suppliers, distribution centers and stores to reliably meet demand while minimizing cost

Solutions to these problems are often found by repeatedly testing what-if scenarios– making adjustments in each scenario to mimic various conditions to assess risks and strategies. To expedite this process, specialized software solutions can be leveraged. There are both off-the-shelf solutions tailored to specific types of optimization problems as well as commercial and open-source optimization solvers that allow for customized mathematical models to address a broad array of business needs. At the heart of all of these solutions are optimization algorithms designed to efficiently find an optimal solution without having to exhaustively enumerate all possible options.

Commercial-grade solvers like Gurobi, along with data and analytics platforms like Databricks, are increasingly being used by businesses to address optimization challenges. These platforms help prepare data inputs and turn solver outputs into actionable applications. In this blog, we will demonstrate how Gurobi and Databricks can work together to solve a simple optimization problem, providing teams with a starting point to tackle similar challenges in their own organizations.

Optimizing a Toy Brick Collection Build

To help us explore how Gurobi and Databricks can be used to solve optimization problems, we’ll start with a simple, illustrative scenario. Imagine you’re a kid (or an adult) and you own the following four Star Wars LEGO® sets:

1. LEGO® Star Wars 75168: Yoda’s Jedi Starfighter (262 pieces)
2. LEGO® Star Wars 75170: The Phantom (269 pieces)
3. LEGO® Star Wars 75162: Y-Wing (90 pieces)
4. LEGO® Star Wars 75160: U-Wing (109 pieces)

Like a lot of folks, you build each set out per the instructions, and when you’re done with that, you disassemble each one, combining the bricks in one large bucket (Figure 1).

The question you have now is, which other official sets could you build from this bucket of bricks? To answer this, we need to clarify four elements of an optimization problem:

• Input parameters - The input parameters define the context for the problem we are trying to solve. In our \example, one input parameter is the number of each type of brick available from our four original sets.
• Decision variables - The decision variables define the choices we have or the decisions we need to make. In this example, the different sets we might build define our decision variables.
• Objectives - Our objectives are the goals we seek to minimize or maximize, represented by a mathematical expression. In this example, we are attempting to maximize the number and size of the sets built while also minimizing the number of left-over bricks following the build-out.
• Constraints - The constraints represent conditions or restrictions that must be met for a proposed solution to be considered valid. In our example, the one constraint is that any set we decide to construct must be complete using the necessary brick parts specified by the official set. In addition, we will constrain our bucket of bricks to hold only the bricks from the four original sets we started with.

With these elements defined, we can now start sorting through potential solutions. With 730 individual bricks in our bucket, we could face more than 10⁷⁵ possible combinations. The fact that there are many identical bricks within each set and more across these sets reduces this number but the resulting number of potential combinations is still overwhelming. We need an intelligent way to navigate the problem space. This is where the solver comes in.

The magic behind the solver is that it can examine the problem (as defined in terms of input parameters, decision variables, etc.) and mathematically explore the problem space to focus on just the solutions that satisfy business rules and improve outcomes. To illustrate this, consider the 730 individual bricks in our bucket. There are no sets to consider that consist of just 1, 2 or 3 bricks, so any iterations that might explore combinations like these can be eliminated from consideration.

By closely examining the problem definition, the solver can tightly constrain the problem space to be explored. The overwhelming number of possible combinations now becomes much more manageable, and through a highly optimized solutioning engine, the remaining outcomes can be rapidly evaluated to deliver the correct answer quickly.

Gurobi and Databricks: Better Together

As more and more organizations consolidate their data assets on Databricks, it’s essential they are enabled to unlock the fullest potential of that data to solve a wide range of business needs. The seamless integration of Gurobi with the Databricks Data Intelligence Platform means that when organizations encounter optimization challenges, they can prepare the data assets in-place without needing to replicate them to another platform. The operations team, familiar with optimization, can then employ the resources of the Databricks environment to solve the problem in a scalable, time- and resource-efficient manner.

With the output of the solver then captured within Databricks, the organization can then integrate the solver’s results into the various operational workflows orchestrated within the environment. And, with access to the integrated model management capabilities of Databricks, these teams can fold their work into enterprise-standard model management and governance practices centered on the platform.

To help organizations get started exploring the use of the Gurobi solver on Databricks, we invite you to take a look at the following sample notebooks, providing access to the step-by-step code behind our toy brick example. Please note that the first two notebooks rely on the solution of small-scale examples that can be solved using the free trial license that Gurobi offers with the installation of its Python API library. The third notebook uses a larger scale model: please contact Gurobi to obtain an appropriate license to run the models in the third notebook.

To understand how organizations can scale out their use of Gurobi with Databricks, we also invite you to watch the following webinar from Aimpoint Digital, a market-leading analytics firm at the forefront of solving the most complex business and economic challenges through data and analytical technology. In this video, the folks at Aimpoint Digital examine the technical integration between Databricks and Gurobi in greater detail and explore various ways organizations can combine these technologies to solve a range of business problems.

Finally, we encourage you to come back to the Databricks blog site to review our upcoming blog on Assortment Optimization which will build on the concepts illustrated here to tackle a more complex, real-world scenario of interest across many retail and consumer goods organizations.

Download the notebooks