the inverse is the correct path.
you might want to work with 4x4 matrices instead of carrying R and t individually.
to simplify the rotation between camera and board, you might wanna define the coordinate axes on the board to be somewhat like the camera’s. if your red is X and green is Y… then maybe define X to point right on the board, and Y down instead.