Thank you all (@Eduardo and @crackwitz ) for your help. I have solved my problem, My main problem was in data collection. The eye-hand calibration method is very sensitive to the target and gripper poses. For all who might be interested in this topic in future, here is my final code for eye-hand calibration:
def calibrate_eye_hand(R_gripper2base, t_gripper2base, R_target2cam, t_target2cam, eye_to_hand=True):
if eye_to_hand:
# change coordinates from gripper2base to base2gripper
R_base2gripper, t_base2gripper = [], []
for R, t in zip(R_gripper2base, t_gripper2base):
R_b2g = R.T
t_b2g = -R_b2g @ t
R_base2gripper.append(R_b2g)
t_base2gripper.append(t_b2g)
# change parameters values
R_gripper2base = R_base2gripper
t_gripper2base = t_base2gripper
# calibrate
R, t = cv2.calibrateHandEye(
R_gripper2base=R_gripper2base,
t_gripper2base=t_gripper2base,
R_target2cam=R_target2cam,
t_target2cam=t_target2cam,
)
return R, t