Detecting flat surface object but the depth return is not consistent

Hi, i have a stereo camera system capture an object put on table (which is flat surface) but the depth return showns is not consistent
From the colormap we can see that the object color is not consistent, however it is put flat on table, how can i get the constant depth from the angled stereo camera system??

i have done asymmetrical circle grid calibration,stereo calibration, stereo rectify, InitUndistortRectifyMap,remap and finally using stereoSGBM for the output.

Thanks for your help, sincerely need help