Anyone experimenting with real-time projection mapping and OAK-D? With a fast enough projector (e.g. a “gaming” projector), it might be possible to do fun things like this:
I always thought real-time face mapping was pretty cool, too:
1 Like
That’s probably possible. The question is the quality you need.
The first application is in fact pretty basic, it just tracks a highlighted object.
The second, in contrast, is probably very complicated.
Check the existing solutions and articles for kinect projection mapping. They probably also apply for other 3D cameras.
You’ll also need fast rendering to get the images out to the projector in real time.
Small note: probably 240Hz vs 60Hz effect the first video was edited. As youtube videos have a 30Hz framerate, there should be no difference between the two parts. So you might probably get good results with 60Hz projector.
1 Like
what you need is low latency, or else the output can’t keep up with the scene.
the refresh rate of both the projector and camera merely pose a (soft) upper bound on the latency of these devices.
I say “soft” because the hard limit is exposure time, which is limited to time 1/fps (because cascading exposures spanning several frames… haven’t heard of that), and soft means all the digital processing in software and hardware might introduce arbitrary latency on top, possibly several full frames worth of time.
you might even have to predict the scene into the future for a fraction of a second.
How easy?? I would love to figure out the active dynamic projection onto the highlighted object OAK-D workaround!
please elaborate. why did you choose that word?
“Workaround” - a solution for incorporating the data into my projection mapping software for real-time tracking
We can give you advice concerning computer vision related questions, but not about using some camera data in some other third party software.
You should probably check the projection software forums or extensions if there is support for OAK-D.
Then, you can begin with some simple processing, like thresholding the depth image to segment the foreground, clean up using some morphology operators and use this image for rendering.
Then, you can go further by adding some real time detection code in the camera to detect and track certain objects. See this tutorial.
1 Like
I tested recently the Oak-D camera. It would be a poor choice for projection mapping.
- It uses stereo image pairing to estimate depth data. In a dark environment with lot of black areas (as you’ll probably use the mapping) this is difficult and you’ll probably have lot of errors in the depth image.
- The depth image latency is higher than other RGB-D cameras (as stereo pairing is computationally expensive).
- Depth resolution degrades with distance
Active stereo cameras (TOF) are probably a much better option for this application. They “see” in the dark too, have a low latency and a good depth resolution. You might try a Realsense camera (the D455 has a 6m range).
1 Like
anything stereo calculates disparity. that includes “projector” types (kinect, realsense, throwing a speckle-like pattern into the scene, giving it texture). depth (res) is inversely proportional to disparity (res). disparity maps are calculated using block matching (correlation), which is computationally expensive. no way around it. that’s how our eyes work too. projector-based stereo brings its own light, so it’s robust to scene lighting. stereo using just cameras requires the scene to be lit tolerably. some algorithms are dumber than others, requiring more computation. anything running on video has the chance to be a lot cheaper than for history-less situations (just a pair of pictures) because one can assume changes between frames to be small, allowing faster convergence.
TOF sensors actively emit light pulses and measure time of flight (round trip time) of the light. that makes their resolution constant over the distance. that is physically/electronically challenging.
I too would recommend that you go with a TOF sensor.