A framework for on-the-fly training and real-time multi-view streaming of sports events in 3D

What is it about?

We present LiveSplats, a framework that transforms any live sports broadcast into a fully interactive 3D experience. Instead of watching from a fixed camera, fans can instantly jump to any angle: sideline, aerial, even right behind the basket. And replay key moments as if they were on the court. Everything happens in real time, so there’s no waiting for special edits or pre-rendered shots. Behind the scenes, we capture multiple video feeds, stitch them together with our fast 3D reconstruction engine, and stream the result with minimal delay. The action looks and feels like you’re there in person: players’ movements are lifelike, lighting adapts naturally, and the stadium environment surrounds you in full depth. To help researchers and developers build on this innovation, we’re also releasing BASKET-Multiview, a massive synthetic basketball dataset. It includes high-definition video, camera calibration, player masks, depth maps, animated 3D meshes and even controlled pose noise. Whether you’re testing new algorithms or creating fresh fan experiences, BASKET-Multiview provides everything you need to push live 3D streaming to the next level.

Featured Image

Photo by Ryan on Unsplash

Photo by Ryan on Unsplash

Why is it important?

Existing VR sports experiences are limited to pre-recorded content, with true live 3D streaming still out of reach. Our work is the first to present a complete, scalable system for capturing, reconstructing, and streaming sports in real time from multiple camera views. This enables immersive, interactive experiences where users can freely explore the game as it unfolds. Our architecture is designed for speed and scalability, achieving near real-time reconstruction of fast-paced, human-centric scenes like basketball, with performance that improves linearly with added compute. To drive progress in this area, we also introduce the first large-scale, multi-view basketball dataset, complete with high-fidelity visuals and rich ground truth, to serve as a benchmark for future research in 3D sports streaming.

Perspectives