Olin Award-winning research: Sifting inhuman options for the human choices

Dennis Zhang and Jake Feldman’s research started as a debate—a little like the Reese’s candy commercial pitting peanut butter lovers against chocolate lovers.

In the end, they combined the ingredients to come up with something bigger.

In the case of their Olin Award-winning research, however, when Feldman got “customer choice modeling” in Dennis Zhang’s “machine learning” and Zhang got “machine learning” in Feldman’s “customer choice modeling,” the result was 28% higher revenue in a week’s time for Chinese online retail giant Alibaba.

The result came about because the pair collaborated to create a new customer choice algorithm designed to better populate six available slots for products in online stores hosted by Alibaba. Those six products appear because the platform instantly crunches millions of variables to display options with the highest likelihood of driving a sale.

Zhang and Feldman, both assistant professors of operations and manufacturing management at Olin, thought the platform could do better—but not before they realized they could combine their disparate approaches into a new mathematical model for presenting product choices to customers. The pair partnered with Alibaba researchers on their working paper, “Taking Assortment Optimization from Theory to Practice: Evidence from Large Field Experiments on Alibaba,” under consideration by a top journal.

“I worked at Google as a machine learning scientist,” Zhang said. “I was a true believer of machine learning, which can basically crack most problems. But Jake is a hardcore believer of (customer choice) optimization.”

In order to convince each other, we started working on this project by combining machine learning with optimization.  

Dennis Zhang

For Alibaba and other online retailers, the problem goes something like this: Among thousands of available products, which ones should the platform recommend to maximize revenue and give customers the most useful choices? Alibaba had relied on machine learning to quickly gauge a visitor’s past purchasing history, age, location, the history of similar customers and a million other variables to come up with a selection of choices to display.

But while machine learning can instantly weigh those variables and display customized product choices, it’s a poor tool for providing wider variety, often displaying products that overlap with or cannibalize other products.

As Feldman describes it, the problem is two-tiered: The first tier is about estimating what customers might like. Machine learning is good at that. The second is optimizing the choices based on what other choices are available and who the customer is—a weak spot for machine learning.

The machine learning-based system might present shoppers with two very similar shirts, or shirts with very different price points—products that don’t offer the right variety or might undermine the retailer’s ability to maximize revenue. It’s particularly troubling when one product is not quite right, but the shopper isn’t given a suitable substitution option.

So, Feldman turned to mathematical models for optimizing customer choices from half a century ago. “They’re not as robust as these machine learning models,” he said. “But what that buys us is we can then solve a more sophisticated optimization problem.”

The researchers tested their new algorithm against the traditional machine learning model Alibaba had used. In a weeklong experiment in March 2018, watching 14 million customers on Alibaba-owned shopping sites, the new combination model showed 28% higher revenues—or nearly $22 million. The results were so conclusive, Alibaba adopted the new product selection model.

The pair agrees that the next step in their research—already underway with Alibaba—is tweaking the model to include data about products the shopper has shown interest in by clicking on them without a purchase. The new model also breaks when more than one product choice is appealing to the customer—another area ripe for further study.

But for now, the pair is very happy with the results of this first collaboration. The results, Zhang said, “seem very inspiring to me because it shows my field and Jake’s field are actually important in the real world of business.”