Incremental structure from motion (adding N views to an existing two-view reconstruction)

I am currently developing a project based on the structure from motion pipeline. My approach is as follows. For each image within a sequential set of images, I extract their features using SIFT. I group the images in pairs (0-1, 1-2, 2-3 …) and perform feature matching using FLANN. I then compute F and filter the outliers, compute E and filter the outliers, and compute R and t using cv::recoverPose. Finally, I perform the triangulation phase. I am able to create a point cloud for each image pair. I plotted each point cloud (0-1, 1-2 …) and I get the expected output. My question is, how do I merge them?
I was thinking that I could approach the situation in another way and use cv::solvePnPRansac but I am not sure how to proceed. Any ideas?

    for (int i = 0; i < images.size(); i++)

    // create image pairs (i, i+1)
    for (int i = 0; i < images.size() - 1; i++)
        ImagePair pair(images[i], images[i + 1]);

    for (int i = 0; i < pairs.size(); i++)
        pairs[i].triangulate(K, i);

Initially, my pipeline was as follows but I got the wrong results…

void SfmReconstruction::triangulation()
    cv::Mat R1 = cv::Mat::eye(3, 3, CV_64F);
    cv::Mat t1 = cv::Mat::zeros(3, 1, CV_64F);

    cv::Mat P1(3, 4, CV_64F);
    cv::hconcat(R1, t1, P1);
    P1 = this->K * P1;

    for (size_t i = 0; i < this->frames.size(); i++)
        cv::Mat points_3d;




        cv::Mat R2 = frames[i].get_R();
        cv::Mat t2 = frames[i].get_t();

        cv::Mat P2(3, 4, CV_64F);
        cv::hconcat(R2, t2, P2);
        P2 = this->K * P2;

        cv::Mat points_4d;
        cv::triangulatePoints(P1, P2, frames[i].get_image1_good_matches(), frames[i].get_image2_good_matches(), points_4d);
        cv::convertPointsFromHomogeneous(points_4d.t(), points_3d);

        R1 = R2.clone();
        t1 = t2.clone();
        P1 = P2.clone();