Why it can not calibrate with found corners and real coordinates?

I would like to ask some questions about calibration.
The background is that I set four chessboards with different position in the same view. When I calibrate camera I can find all corners p(uij, vij) (input imagePoints ) of four chessboards in the same image.Since I have known the four chessboards positons, I can get the real points coordinates P(xij,yij,zij)(input objectPoints) of each chessboard(where i=0~3, j=0~patternSize). But when I use the calibrateCamera() to calibrate, the program proceed with error util I set all zij=0.I am confused why I have the real points coordinates and corners coordinates but can not calibrate?

are you trying to calibrate one picture containing four boards at the same time?

that violates the assumptions about how calibration works.

calibration works with exactly one board in view, and it should be the same board in all views.

 Yes,I'm trying to calibrate one picture containing four boards at the same time. 
 I put four boards(00, 01, 10, 11)  at the four quadrants( 2nd quadrant, 1st quadrant, 3rd quadrant, 4th quadrant  ).The four boards(00, 01, 10, 11) have the same pattern size and chessboard size for each, and their positions are very regular rotation/translation as follows:

屏幕截图 2021-05-27 104213
So I can exactly get the found corners and real coordinates. And I thought it’s equivalent to one board in four views.For example of board01, I can find the corners vector p01 and compute real coordinates vector P01. In order to obey the opencv assumptions, I use board01 plane real coordinates, and then I can get the intrinsic cameraMatrix and distCoeffs. In this way I can get the final cameraMatrix and distCoeffs.
I am not sure it is right way to compute the final cameraMatrix and distCoeffs, it seems that cameraMatrix is right (compared to my manual computation results).
Furtherly, I guess the current opencv calibratioan API use Z=0 board plane coordinates assumption and can not support Z!=0 condition, but why it do not add calibratioan API of Z!=0 condition(the real coordinates) ?