Popularity matters more than compatibility: Study uncovers bias in algorithms used by online dating platforms

A new study has found that algorithms used by online dating platforms have popularity bias—meaning that they recommend more popular, attractive users over less popular, less attractive users. Researchers at Carnegie Mellon University and the University of Washington published their findings in the journal Manufacturing & Service Operations Management.

They evaluated data from over 240,000 users of a major online dating platform in Asia over three months. They found that a user's chance of being recommended by the platform's algorithm increased significantly as their average attractiveness score went up. This suggests the algorithm is biased toward recommending users who are more popular or considered more attractive on the platform.

"Online dating has grown rapidly—especially during the COVID-19 pandemic," noted Soo-Haeng Cho, IBM Professor of Operations Management and Strategy at Carnegie Mellon's Tepper School of Business, who coauthored the study. "Even though dating platforms allow users to connect with others, questions regarding fairness in their recommendation algorithms remain."

Users join online dating platforms to find matches, but the companies creating the platforms also need to generate revenue. Companies make money through ads, subscriptions, and in-app purchases. For this reason, platforms may seek to keep users engaged on their platforms rather than maximizing their chances of finding the perfect person.

The researchers built a model to analyze the incentives for platforms to recommend popular users more frequently when their goal is to maximize revenue or maximize matches. In their model, they used the unbiased approach (which is when popular and unpopular users find equal chances to be recommended to others) as their benchmark for fairness to compare popular and unpopular users' matching probabilities.

Their analysis shows that unbiased recommendations tend to result in significantly lower revenue for the dating platform and fewer matches. This is because popular users help the platform generate more revenue by boosting users' engagement (through more likes and messages sent). In addition, popular users help the platform generate more successful matches as long as they do not become so selective that they are seen as being out of reach to less popular users.

The study also found that popularity bias may be low when a platform is in the early stage of growth because a high match rate can help build a platform's reputation and bring in new users. But, as the platform matures, its focus may shift to maximizing revenues, leading to more popularity bias.

"Our findings suggest that an online dating platform can increase revenue and users' chances of finding dating partners simultaneously," explains Musa Eren Celdir, who was a Ph.D. student at Carnegie Mellon's Tepper School of Business when he led the study. "These platforms can use our results to understand user behavior and they can use our model to improve their recommendation systems."

"Our work contributes to the research on online matching platforms by studying fairness and bias in recommendation systems and by building a new predictive model to estimate users' decisions," says Elina H. Hwang, Associate Professor of Information Systems at the University of Washington's Foster School of Business, who also co-authored the study.

"Although we focused on a specific dating platform, our model and analysis can be applied to other matching platforms, where the platform makes recommendations to its users and users have different characteristics."

The researchers suggest that online dating platforms be more transparent with users about how their algorithms work. They also noted that more research is needed on how to balance user satisfaction, revenue goals and ethical algorithm design.