So when creating a new environment using conda, I installed pytorch and torchvision using the following command.
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
Later when trying to install OpenCV using conda install opencv , it downgrades the torchvision package from 0.17(or a more recent one) to 0.15a which becomes incompatible with latest version of pytorch==2.2.X. I dont know who is responsible to fix this part.
Alternate workaround I found:
Installation of Opencv using Conda install is forcing this downgrade to incompatible versions, the only workaround I could find was to activate the environment and install opencv using pip install.