What is it about?
Normal cameras record only red, green, and blue. A hyperspectral camera separates light into many narrow wavelength bands, allowing it to reveal differences between materials that may look identical in a normal image. The challenge is that dividing light across position, time, and wavelength leaves very few photons for each measurement. As a result, hyperspectral video is often slow, noisy, or limited in detail. We developed Hi-SPAD, a camera that uses sensors capable of detecting individual photons. These sensors operate extremely quickly and avoid the electronic readout noise that limits conventional cameras when little light is available. Our laboratory prototype captures hyperspectral video at 26 frames per second in 123 wavelength bands spanning visible and near-infrared light, using ambient illumination. We demonstrate how these videos can be used to distinguish materials, change the apparent lighting of a scene after recording, and reproduce how different camera models would see the same scene.
Featured Image
Photo by David Clode on Unsplash
Why is it important?
Existing hyperspectral cameras usually trade one capability for another: they may capture many wavelengths but operate slowly, capture video but lose spatial or spectral detail, or depend on active illumination and complex optical encoding. Hi-SPAD demonstrates a different approach based on high-speed, photon-counting sensors. By collecting rapid binary measurements and processing them with statistical models designed specifically for single-photon data, the system preserves useful spectral information even when only a handful of measurements are available for each video frame. This work combines a working camera prototype with theoretical analysis and new reconstruction and classification methods. It could make hyperspectral sensing more practical for moving scenes and time-sensitive applications, including rapid material identification and post-capture analysis of lighting and camera response. More broadly, it shows a path toward cameras that record not only how a scene looks, but also richer information about its materials and how they interact with light.
Perspectives
This project grew from a question that initially seemed counterintuitive: can extremely sparse, one-bit photon measurements contain enough information to produce useful hyperspectral video? The result I find most exciting is that they can—provided that we model the sensor’s nonlinear statistics rather than treating it like an ordinary camera. Seeing the theory carry over from simulations to a laboratory-built system and real dynamic scenes was especially rewarding. I hope this work encourages researchers to view single-photon cameras as more than devices for low-light photography or depth sensing. They can also provide a platform for recording richer dimensions of light. Important challenges remain, including rolling-shutter distortion, stray light, limited sensor resolution, and real-time reconstruction, but these challenges also point toward clear opportunities for improved hardware, programmable scanning strategies, and reconstruction algorithms.
Haejoon Lee
Carnegie Mellon University
Read the Original
This page is a summary of: Hi-SPAD: Video-Rate Hyperspectral Imaging and Inference with Single-Photon Cameras, ACM Transactions on Graphics, July 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3829360.
You can read the full text:
Resources
Contributors
The following have contributed to this page
