use DIS optical flow, it’s faster and better than LK.
optical flow for a camera moving forward/backward will have vectors pointing towards the vanishing point, which may or may not be in the center of the view. it will be the point towards which the camera moves.
you should polar-warp those vectors instead of the picture. polar-warping the picture requires interpolation. warping the vectors does not.