Incremental structure from motion (adding N views to an existing two-view reconstruction)

C++
1

I am currently developing a project based on the structure from motion pipeline. My approach is as follows. For each image within a sequential set of images, I extract their features using SIFT. I group the images in pairs (0-1, 1-2, 2-3 …) and perform feature matching using FLANN. I then compute F and filter the outliers, compute E and filter the outliers, and compute R and t using cv::recoverPose. Finally, I perform the triangulation phase. I am able to create a point cloud for each image pair. I plotted each point cloud (0-1, 1-2 …) and I get the expected output. My question is, how do I merge them?
I was thinking that I could approach the situation in another way and use cv::solvePnPRansac but I am not sure how to proceed. Any ideas?

    for (int i = 0; i < images.size(); i++)
    {
        images[i].compute_kps_des(FeatureDetectionType::SIFT);
    }

    // create image pairs (i, i+1)
    for (int i = 0; i < images.size() - 1; i++)
    {
        ImagePair pair(images[i], images[i + 1]);
        pairs.push_back(pair);
    }

    for (int i = 0; i < pairs.size(); i++)
    {
        pairs[i].match_descriptors(FeatureMatchingType::FLANN);
        pairs[i].compute_F();
        pairs[i].compute_E(K);
        pairs[i].compute_Rt(K);
        pairs[i].triangulate(K, i);
    }

Initially, my pipeline was as follows but I got the wrong results…

void SfmReconstruction::triangulation()
{
    cv::Mat R1 = cv::Mat::eye(3, 3, CV_64F);
    cv::Mat t1 = cv::Mat::zeros(3, 1, CV_64F);

    cv::Mat P1(3, 4, CV_64F);
    cv::hconcat(R1, t1, P1);
    P1 = this->K * P1;

    for (size_t i = 0; i < this->frames.size(); i++)
    {
        cv::Mat points_3d;

        frames[i].match_descriptors(FeatureMatchingType::FLANN);

        frames[i].compute_F();

        frames[i].compute_E(this->K);

        frames[i].compute_Rt(this->K);
        cv::Mat R2 = frames[i].get_R();
        cv::Mat t2 = frames[i].get_t();

        cv::Mat P2(3, 4, CV_64F);
        cv::hconcat(R2, t2, P2);
        P2 = this->K * P2;

        cv::Mat points_4d;
        cv::triangulatePoints(P1, P2, frames[i].get_image1_good_matches(), frames[i].get_image2_good_matches(), points_4d);
        cv::convertPointsFromHomogeneous(points_4d.t(), points_3d);

        R1 = R2.clone();
        t1 = t2.clone();
        P1 = P2.clone();
    }
}