From game footage to great footage with computer vision and other artificial intelligence tools

How much cooler would basketball footage be if software could automatically shine a spotlight on key players? How much easier would it be to understand plays if arrows could indicate movement before the players actually move? How much improvement would be possible if one could see how distance from a basket affects shot trajectory?

All of these features are now possible thanks to SportsBuddy. A collaboration between the Harvard Visual Computing Group (VCG) at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and the Harvard men's basketball team, SportsBuddy uses computer vision and other artificial intelligence tools to augment basketball game footage, improving the viewer's experience with a range of special effects such as spotlights on specific players, arrows to indicate player movement, and sparkling or flaming basketballs to signify a successful play.

Professional sports leagues like the NBA already have the technology to produce highlight reels like this, but that's not always the case for collegiate or recreational leagues. The computer vision techniques in SportsBuddy enable it to work without advanced cameras or motion sensors, meaning a device as simple as a smartphone camera could be used to create a high tech highlight reel. And by combining computer vision with the right natural language processor interface, a user could simply type in the players and effects they want, and SportsBuddy does the rest to produce the highlight.

"Our ultimate goal is to convey the data in a more accessible format and provide deeper insights to the audience," said Tica Lin, co-lead developer for SportsBuddy and a Ph.D. candidate in computer science in the VCG. "Without any sensing technologies, we can directly derive data from the video and embed back information to deliver insights."

Lin, who worked in data analytics and visualization for the Philadelphia 76ers before joining SEAS, and her team have produced augmented highlight reels throughout the 2023–24 basketball season. Recent videos have even featured narration by a Harvard coach, with on-screen effects matching his descriptions of plays.

"We started with racket sports like tennis, which was easier than basketball because there's only two players with no overlap," said Zhu-Tian Chen, a postdoc in the VCG. "Once we were ready to move on to more complicated sports, we decided on basketball because it strikes a balance between the number of players and their overlap."

As the number of videos has increased throughout the season, first-year forward Luca Ace-Nasteski says he's noticed growing interest in being highlighted among his teammates.

"It's a cool idea," he said. "I hadn't seen it before, except in the NBA."

Lin is also working on VCG's other collaboration with the Harvard Crimson, SportsXR, which uses AR technology to improve data analytics and visualization in athletics.

"Sports turned out to be a perfect application domain for AR/VR," said Lin, who studied augmented and virtual reality (AR/VR) technology as a master's student at the Georgia Institute of Technology. "When athletes and coaches work out on the court, they're far away from the traditional desktop displays that most visualizations are designed for. AR/VR is a really appealing new medium to bring this data closer to the athletes and coaches."

VCG and the Neuromotor Control Lab at SEAS collaborated with Harvard's men's and women's basketball teams to test SportsXR. Players wore AR/VR goggles that could analyze their shooting trajectory as they practiced free throws, and would then adjust their shooting form to try to bring the trajectory in line with the ideal arc of the ball from their hands to the basket.

SportsXR predates SportsBuddy by several years, but HVCG director Hanspeter Pfister, An Wang Professor of Computer Science, said that technology is still 10–20 years away from becoming a viable commercial product. SportsBuddy could be ready much sooner, and the goal is to continue refining the user interface to spin it off into a company.

"Zhu-Tian and Tica are fantastic researchers," Pfister said. "They came up with a lot of the new ideas that you're seeing. I always say research is there to learn from applications, and applications are there to learn from research. We had a research idea focused on extended reality, which is still 20 years away, but now we have a spin-off, SportsBuddy, which has very practical applications and is more focused on an actual problem teams try to solve.