Enhancing cooperation of AI and human planners

Supply chain planners have the opportunity to change forecasts produced by the self-learning systems using artificial intelligence. Does this add value? And which forecast can be relied on more—the software's or the human's? In a state of the art study, KLU researchers Naghmeh Khosrowabadi, Prof. Kai Hoberg and Prof. Christina Imdahl (Eindhoven University of Technology) are getting to the bottom of these questions. According to their study, human intervention—on average—did not increase the accuracy of the forecasts.

For the study, the team analyzed data of 30 million forecasts for each unit of product, per store on each day from a leading AI provider and major European grocery retailer.

Planners' optimism leads to more inaccurate forecasts

The results show that planners—on average—do not contribute to the accuracy of the forecast. "Instead, planners even tend to overcompensate effects like weather or a discount that have already been considered by the AI system," says Khosrowabadi. In the study, only 50 percent of human interventions led to improved results.

A closer look at the data additionally reveals that around five percent of AI-generated forecasts were adjusted by supply chain planners. "We wanted to know why planners decided to adjust the AI-generated forecasts," explains Naghmeh Khosrowabadi. "Our results show that characteristic of the product like price, freshness or discounts are key drivers for the frequency of planners' adjustments to AI forecasts."

If, for example, the AI system is giving a forecast for a very expensive product, planners tend to pay more attention and are more likely to intervene, e.g. by adjusting the forecast. "Besides, our results show that large increases from the AI forecast, for example when the human forecast for items to be sold on a given day in a specific store is two times greater than the AI forecast, are more frequent but also often inaccurate. Too much optimism on the side of the planners seems to be an issue here," says Prof. Kai Hoberg. Decreases from the AI-forecast, on the other hand, were less likely but more accurate.

Improving the cooperation of human planners and AI

"Human planners will rightful continue to play an important role in AI-enabled forecast processes," says Prof. Hoberg, "In certain cases human planners have knowledge that is not accessible to an AI system, e.g. local events or competitor actions, that enables them to increase the odds for a better forecast to more than 70%. That is why we need to enhance the cooperation of planners and AI."

For this, the team recommends more exchanges between retailers and AI providers: the better planners understand how the system makes its forecasts, the easier it is for them to decide when to intervene. "Using the results of our study, companies may save money and time," promises Khosrowabadi, "The key is to help planners to decide when they need to intervene—and when the system is fine on its own and they may concentrate on other tasks."

The study involved Naghmeh Khosrowabadi as part of her doctoral thesis, Prof. Dr. Kai Hoberg and Prof. Dr. Christina Imdahl from KLU. They analyzed data from 30 million SKU store-day level forecasts from a leading AI provider and a major European retailer. Data on additional variables such as products, weather or holidays were also considered.

The work is published in the European Journal of Operational Research.