Java Structure From Motion help, please

Android/Java
1

Hello, OpenCV friends.

I’ve got an open source java robot simulation + control project called Robot Overlord.

I’ve mounted a camera on one of the robot arms. I’d like to use structure-from-motion to scan the local environment and build a mesh so that I know the collision boundaries. It would be nice if I could also calibrate the arm from the SFM camera position data.

To date the Maven OpenCV repos I’ve found don’t appear to handle java well. Maybe they were setup for one part of OpenCV …but not the bit I want to work with.

Then again maybe I’m doing it wrong. I haven’t found a great OpenCV java tutorial that explains what I want. opencv-java-tutorials.readthedocs.io got as far as camera calibration and has yet to write SFM tutorials or other advanced features.

What do? Would a bounty help? To whom would I make that proposal?

2

just saying, SFM will give you a point cloud, not a mesh
also, “depth-from-a-single-image” dnns have improved a lot lately,
look at that, too !

1 Like
3

Even a point cloud would be a huge step up from what I have. Please, how do I make this work in Java?

4

there is an sfm module in opencv_contrib, but no java bindings for that, so far. (python/c++ only) also:tutorials

those are a 3rdparty effort, not maintained by opencv devs, same problem with the tutorial you mentioned (updates there stopped 2016 ?)

to get anything advanced done, you’ll have to build opencv libs from src, anyway.

sorry for all the bad news, but this is advanced stuff, and support for java is sparse

5

in the meantime, maybe i can get you interested in the dnn idea ?
(it would work with no “extra” modules installed)

girl
girl_d

sorry, c++ code:

    string nn ="g:/data/dnn/DE_resnet_unet_hyb.onnx";
    dnn::Net net = dnn::readNet(nn);

    string ipath = argc>1 ? argv[1] : "c:/data/out/depth/girl.png";
    Mat img = imread(ipath);
    imshow("I", img);
    waitKey(1);

    Size size = Size(320,256);
    Mat blob = dnn::blobFromImage(img, 1.0/255, size, Scalar::all(127.0), false, true);
 	net.setInput(blob);
 	Mat res = net.forward();

    Mat depth = res.reshape(1, res.size[2]); // by height
    normalize(depth, depth, 0, 1, NORM_MINMAX, CV_32F);
    pyrUp(depth,depth);

    imshow("D",depth);
    waitKey();

to get the model:

#!/bin/bash
fileid="14n3cTmaKGJHIhL0sBRaByuyPBDyntHZm"
filename="unet.onnx"
curl -c ./cookie -s -L "https://drive.google.com/uc?export=download&id=${fileid}" > /dev/null
curl -Lb ./cookie "https://drive.google.com/uc?export=download&confirm=`awk '/download/ {print $NF}' ./cookie`&id=${fileid}" -o ${filename}
6

At some point there are enough “advanced” users that it makes sense to have a maintainer. Does anyone volunteer as tribute? I would gladly join your Patreon and support your work.

7

to do what, exactly ?

8

To ensure a working java interface for opencv, including advanced features and tutorials for said features.

1 Like
9

i whish it’d be so. but there arent. that’s why the sad state of it.