Maybe it’s easier to ignore solvePnp and find a special solution instead.
Could be involved:
- Translate your object by the known translation.
- For each of your individual markers there are 2 possible rotations (using 2 of the 3 rotation’s degrees of freedom) so that the marker projects to that pixel.