How to convert mat to 3D matrix

please try NOT to write code like this. it is slow, error-prone, and defeats the purpose of a matrix oriented library.

explain, what you are trying to achieve in the end, then we can help you to make use of opencv’s builtin warping functions.