UPDATE
I was using an export script that comes with YOLOv5 from Ultralytics to create the ONNX file. I just cloned the latest version of YOLOv5 from git. Problem went away.
On the bright side I did learn a lot by building a debug version of OpenCV and stepping through the code 