Tracking a soccer field while camera pans/moves

I’m currently using object detection to detect football players on a football pitch. From the x,y I receive I then want to transform them and use cv2.circle to plot them on a different image (a smaller birdseye view of the pitch) for anaysis. I can do this currently but it’s mainly manual as the camera position is not static. So, I take the coods of the penalty box on the real image, then take the coords of the penalty box on my birdsye image then pass them through getPerspectiveTransform which in essence give me the new coords to plot. It works brilliantly! However, as soon as the camera moves it changes perspective and the players are no longer mapped onto the birdseye image in the same place. Can anybody help me or point me in the right direction to solve this issue?