Thanks! I remember I’ve seen the arucoBoard implementation for similar task somewhere, but haven’t decided to try it yet. Even though it should work, I will probably still try to make this barebones, have to dig into the 4x4 transformation matrices though.
I tried using this topic on stackoverflow, there’s a nice function which first does the translation and then rotation, just as you mentioned. But it actually did the same as the regular vector subtraction (or maybe I misunderstood how to use it) 
Btw I used SciPy Rotation to rotate rvec and it worked nicely. Sadly there’s no implementation for translation vector.