Whilst it could be anything the most obvious reason for this is that you are not consuming the frames as fast as they are produced. In that case your OS will have to discard data when its buffer is full. Because you are using h264 where each frame is “not” encoded seperately you encounter decoding errors when this happens which affect multiple frames.
The best solution is to decode in a seperate thread and drop the decoded frames which you cannot process.