Cannot capture stream when written from another thread [OpenCV GStreamer C++]

I have a program that captures video from USB camera, process and stream to rtsp udp. I am using OpenCV with Gstreamer.

When I use the main thread to write out the frames, I can capture it with no problem using gst-launch. However, when I create another thread to do the writing out the frame, nothing happens with gst-launch. I know the other thread is running because I am able to “imshow” the frames in that thread. Also, I am sure that the writer is open since I checked it before writing.

Writer pipeline : appsrc ! videoconvert ! x264enc ! rtph264pay ! udpsink host= port=5015
Receiver: gst-launch-1.0 udpsrc port=5015 ! queue ! "application/x-rtp, media=(string)video, encoding-name=(string)H264, framerate=30/1" ! rtph264depay ! decodebin ! videoconvert ! autovideosink