Need some help with CUDA Optical Flow in Python

Python
,
1

There is no Python documentation available for the CUDA optical flow modules. I have been trying to obtain the Brox optical flow and used some workarounds for bugs I encountered.
However, the output doesn’t make any sense.
If someone could point out any mistakes in my code, I’d be very grateful.

import cv2
import numpy as np

#fresh start
print(cv2.cuda.getCudaEnabledDeviceCount())

vidPath = 'C:/Users/Aditya/Downloads/JESTER DATASET/20bn-jester-v1/1/%5d.jpg'
lr = 0.05
check_res = False
frame_device = cv2.cuda_GpuMat()

#taking input, pass 0 to use webcam data
cap = cv2.VideoCapture(vidPath)

bgmog2_device = cv2.cuda.createBackgroundSubtractorMOG2()
brox_of = cv2.cuda_BroxOpticalFlow.create(0.197, 50.0, 0.8, 10, 77, 10) 
#(alpha ,gamma ,scale ,inner_iterations ,outer_iterations ,solver_iterations)

def bgmogCuda(frame,lr,store_res=False):
    frame_device.upload(frame)
    fg_device = bgmog2_device.apply(frame_device,lr,cv2.cuda.Stream_Null())
    fg_host = fg_device.download()
    if(store_res):
        gpu_res.append(np.copy(fg_host))
    return fg_host
        
gpu_res = []

gpu_prev = cv2.cuda_GpuMat()
gpu_frame = cv2.cuda_GpuMat()

ret, frame = cap.read()
frame = np.float32(frame)/255.0
gpu_prev.upload(frame)
gpu_prev = cv2.cuda.cvtColor(gpu_prev, cv2.COLOR_BGR2GRAY)
print("Type prev = ",gpu_prev.type())

h, w = frame.shape[:2]
temp = np.zeros((h, w), np.float32)
temp = gpu_prev.download()

cv2.imshow('temp', temp)
gpu_v = cv2.cuda_GpuMat(gpu_prev.size(), cv2.CV_32FC1)

#loop frames
while(1):

    ret, frame = cap.read()
    
    if not ret:
        cap.release()
        break
        
    #bgsgm
    fgmask = bgmogCuda(frame, lr, store_res=check_res)
    kernel = np.ones((3, 3), np.uint8)
    erode = cv2.erode(fgmask, kernel, iterations = 1)
    cv2.imshow('bgmog', erode)
    
    #brox optical flow opencv cuda
    frame = np.float32(frame)/255.0
    gpu_frame.upload(frame)
    
    if(gpu_prev.type() != 5):
        gpu_prev = cv2.cuda.cvtColor(gpu_prev, cv2.COLOR_BGR2GRAY)
    gpu_frame = cv2.cuda.cvtColor(gpu_frame, cv2.COLOR_BGR2GRAY)

    gpu_flow = brox_of.calc(gpu_prev, gpu_frame, None)
    gpu_flow_x = cv2.cuda_GpuMat(gpu_flow.size(), cv2.CV_32FC1)
    gpu_flow_y = cv2.cuda_GpuMat(gpu_flow.size(), cv2.CV_32FC1)
    cv2.cuda.split(gpu_flow, [gpu_flow_x, gpu_flow_y], cv2.cuda.Stream_Null())
    
    gpu_magnitude, gpu_angle = cv2.cuda.cartToPolar(gpu_flow_x, gpu_flow_y, angleInDegrees=False)
    
    # set value to normalized magnitude from 0 to 1
    gpu_v = cv2.cuda.normalize(gpu_magnitude, 0.0, 255.0, cv2.NORM_MINMAX, -1)

    res = gpu_v.download()
    
    cv2.imshow("result", res)

    gpu_prev = gpu_frame
    
    if 0xFF & cv2.waitKey(50) == 27:
        break
        
cv2.waitKey(0)
cv2.destroyAllWindows()
2

What errors/bugs are you seeing?

The input/output documentation for all the routines for the BroxOpticalFlow class in python can be found by querying the help.

help(cv2.cuda_BroxOpticalFlow) 
  Help on class cuda_BroxOpticalFlow in module cv2:

 class cuda_BroxOpticalFlow(cuda_DenseOpticalFlow)
 |  Method resolution order:
 |      cuda_BroxOpticalFlow
 |      cuda_DenseOpticalFlow
 |      Algorithm
 |      builtins.object
 |  
 |  Methods defined here:
 |  
 |  __repr__(self, /)
 |      Return repr(self).
 |  
 |  getFlowSmoothness(...)
 |      getFlowSmoothness() -> retval
 |      .
 |  
 |  getGradientConstancyImportance(...)
 |      getGradientConstancyImportance() -> retval
 |      .
 |  
 |  getInnerIterations(...)
 |      getInnerIterations() -> retval
 |      .
 |  
 |  getOuterIterations(...)
 |      getOuterIterations() -> retval
 |      .
 |  
 |  getPyramidScaleFactor(...)
 |      getPyramidScaleFactor() -> retval
 |      .
 |  
 |  getSolverIterations(...)
 |      getSolverIterations() -> retval
 |      .
 |  
 |  setFlowSmoothness(...)
 |      setFlowSmoothness(alpha) -> None
 |      .
 |  
 |  setGradientConstancyImportance(...)
 |      setGradientConstancyImportance(gamma) -> None
 |      .
 |  
 |  setInnerIterations(...)
 |      setInnerIterations(inner_iterations) -> None
 |      .
 |  
 |  setOuterIterations(...)
 |      setOuterIterations(outer_iterations) -> None
 |      .
 |  
 |  setPyramidScaleFactor(...)
 |      setPyramidScaleFactor(scale_factor) -> None
 |      .
 |  
 |  setSolverIterations(...)
 |      setSolverIterations(solver_iterations) -> None
 |      .
 |  
 |  ----------------------------------------------------------------------
 |  Static methods defined here:
 |  
 |  __new__(*args, **kwargs) from builtins.type
 |      Create and return a new object.  See help(type) for accurate signature.
 |  
 |  create(...)
 |      create([, alpha[, gamma[, scale_factor[, inner_iterations[, outer_iterations[, solver_iterations]]]]]]) -> retval
 |      .
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from cuda_DenseOpticalFlow:
 |  
 |  calc(...)
 |      calc(I0, I1, flow[, stream]) -> flow
 |      .   @brief Calculates a dense optical flow.
 |      .   
 |      .       @param I0 first input image.
 |      .       @param I1 second input image of the same size and the same type as I0.
 |      .       @param flow computed flow image that has the same size as I0 and type CV_32FC2.
 |      .       @param stream Stream for the asynchronous version.
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from Algorithm:
 |  
 |  clear(...)
 |      clear() -> None
 |      .   @brief Clears the algorithm state
 |  
 |  empty(...)
 |      empty() -> retval
 |      .   @brief Returns true if the Algorithm is empty (e.g. in the very beginning or after unsuccessful read
 |  
 |  getDefaultName(...)
 |      getDefaultName() -> retval
 |      .   Returns the algorithm string identifier.
 |      .   This string is used as top level xml/yml node tag when the object is saved to a file or string.
 |  
 |  read(...)
 |      read(fn) -> None
 |      .   @brief Reads algorithm parameters from a file storage
 |  
 |  save(...)
 |      save(filename) -> None
 |      .   Saves the algorithm to a file.
 |      .   In order to make this method work, the derived class must implement Algorithm::write(FileStorage& fs).
 |  
 |  write(...)
 |      write(fs[, name]) -> None
 |      .   @brief simplified API for language bindings
 |      .       * @overload

With the specific input/output documentation for the routines you are using being

help(cv2.cuda_BroxOpticalFlow.create) Help on built-in function create: 
create(...)
    create([, alpha[, gamma[, scale_factor[, inner_iterations[, outer_iterations[, solver_iterations]]]]]]) -> retval
help(cv2.cuda_BroxOpticalFlow.calc) Help on method_descriptor: 
calc(...)
    calc(I0, I1, flow[, stream]) -> flow
    .   @brief Calculates a dense optical flow.
    .   
    .       @param I0 first input image.
    .       @param I1 second input image of the same size and the same type as I0.
    .       @param flow computed flow image that has the same size as I0 and type CV_32FC2.
    .       @param stream Stream for the asynchronous version.
1 Like
3

One thing to note is that CUDA Brox optical flow won’t initialize from the input flow. It will treat all video frames as independent. It’d be nice if the call to compute the flow could initialize from the flow image passed in by the user.

4

Hey,

Thanks for your reading my doubt.

I managed to fix it by removing this line
gpu_prev = gpu_frame
and replacing it with
gpu_prev.upload(frame)

I thought I’d leave the issue open just in case any other error arises.

I have a few other queries, though, if you don’t mind answering those.

I checked out Nvidia’s optical flow SDK, and it supposedly gives a much better performance than Brox optical flow.
I tried building the SDK, but Visual Studio is giving me a few errors.
Do you have any experience with that?

Also, does the equivalent algorithm in OpenCV require the SDK to be installed?

5

4 posts were split to a new topic: How to get opencv-contrib binary package with CUDA