Ok, so it seems that my problem with blackness after rotation comes from me not understanding what that offsets were in the cuda::rotate() function, and because of that my image rotated outside of the seen box… Now I rotate using cuda::wrapAffine() and it’s working correctly.
You are right, transferring GPU<->CPU is costly but in my case CPU is working much worse.
Previously (on CPU) my video stream slowed down from 28FPS to 8FPS, now on GPU it’s slowing down to 17FPS
It’s much better but still, I hoped for better performance 
I’m not sure why it’s so bad, I’m working on Jetson Xavier NX. And it’s not so weak of a device to work so poorly I think…?
Do you perhaps have some ideas why it’s that and how can I correct that to have better performance?