Real-time projection mapping?

Anyone experimenting with real-time projection mapping and OAK-D? With a fast enough projector (e.g. a “gaming” projector), it might be possible to do fun things like this:

I always thought real-time face mapping was pretty cool, too:

That’s probably possible. The question is the quality you need.
The first application is in fact pretty basic, it just tracks a highlighted object.
The second, in contrast, is probably very complicated.
Check the existing solutions and articles for kinect projection mapping. They probably also apply for other 3D cameras.
You’ll also need fast rendering to get the images out to the projector in real time.

Small note: probably 240Hz vs 60Hz effect the first video was edited. As youtube videos have a 30Hz framerate, there should be no difference between the two parts. So you might probably get good results with 60Hz projector.

what you need is low latency, or else the output can’t keep up with the scene.

the refresh rate of both the projector and camera merely pose a (soft) upper bound on the latency of these devices.

I say “soft” because the hard limit is exposure time, which is limited to time 1/fps (because cascading exposures spanning several frames… haven’t heard of that), and soft means all the digital processing in software and hardware might introduce arbitrary latency on top, possibly several full frames worth of time.

you might even have to predict the scene into the future for a fraction of a second.