BayerRG -> RGB conversion

Python
1

Hello,

Is there a bug in BayerRG to RGB conversion in OpenCV?

After a lot of research, I have isolated my test case to the following.

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Simple matrix in BayerRG layout of a 6x12 image with all red pixels
I = np.uint8([
    255, 0, 255, 0, 255, 0, 
    0, 0, 0, 0, 0, 0,
    255, 0, 255, 0, 255, 0,
    0, 0, 0, 0, 0, 0,
    255, 0, 255, 0, 255, 0,
    0, 0, 0, 0, 0, 0,
    255, 0, 255, 0, 255, 0,
    0, 0, 0, 0, 0, 0,
    255, 0, 255, 0, 255, 0,
    0, 0, 0, 0, 0, 0,
    255, 0, 255, 0, 255, 0,
    0, 0, 0, 0, 0, 0
          ]).reshape(12, 6)

# Convert BayerRG --> RGB
# NOTE: OpenCV often uses the BGR layout, but here I'm asking for RGB layout explicitly
J = cv2.cvtColor(I, cv2.COLOR_BayerRG2RGB)

# This will show all pixels as blue, which is not what I expect.
plt.imshow(J)

To confirm that the convert image J should indeed be red, feed the same example to MATLAB with the following code, by going to Convert Bayer pattern encoded image to truecolor image - MATLAB demosaic and clicking on “Try This Example” to open interactive coding window.

I = uint8([
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0; ...
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0; ...
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0; ...
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0; ...
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0; ...
255, 0, 255, 0, 255, 0; ...
0, 0, 0, 0, 0, 0
]);

J = demosaic(I,'rggb');

fig=figure, imshow(J); truesize(fig, [48, 24]);

I have tried 2 different versions of cv2: 4.5.1 & 3.4.7. The issue exists in both.

Can anyone please confirm the bug? I

Thanks.

2

Hi,

All pixels are red. opencv uses BGR order

3

@laurent.berger, I know OpenCV uses BGR layout by default, e.g., when decoding a JPG image, but that does not apply here.

Here, I’m explicitly asking for conversion to RGB, as specified by the my COLOR_BayerRG2RGB conversion code (as opposed to COLOR_BayerRG2BGR code). OpenCV should use RGB layout in this case.

4

What’s your opencv version? (print (cv.getBuildInformation())

5

Yep, that might be a bug. I remember also having problems with demosaicing with cvtColor and not getting the expected results; I tought I just mistook the sensor type, si I just tested different combinations until I got the expected results.

1 Like
6

General configuration for OpenCV 4.5.1 =====================================
Version control: 4.5.1

Platform:
Timestamp: 2021-01-02T13:15:20Z
Host: Windows 6.3.9600 AMD64
CMake: 3.18.4
CMake generator: Visual Studio 14 2015 Win64
CMake build tool: C:/Program Files (x86)/MSBuild/14.0/bin/MSBuild.exe
MSVC: 1900

CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (15 files): + SSSE3 SSE4_1
SSE4_2 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (0 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (4 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (29 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2

C/C++:
Built as dynamic libs?: NO
C++ standard: 11
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe (ver 19.0.24241.7)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MT /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MTd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MT /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MTd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /NODEFAULTLIB:atlthunk.lib /INCREMENTAL:NO /NODEFAULTLIB:libcmtd.lib /NODEFAULTLIB:libcpmtd.lib /NODEFAULTLIB:msvcrtd.lib
Linker flags (Debug): /machine:x64 /NODEFAULTLIB:atlthunk.lib /debug /INCREMENTAL /NODEFAULTLIB:libcmt.lib /NODEFAULTLIB:libcpmt.lib /NODEFAULTLIB:msvcrt.lib
ccache: NO
Precompiled headers: YES
Extra dependencies: ade wsock32 comctl32 gdi32 ole32 setupapi ws2_32
3rdparty dependencies: ittnotify libprotobuf zlib libjpeg-turbo libwebp libpng libtiff libopenjp2 IlmImf quirc ippiw ippicv

OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio
Disabled: world
Disabled by dependency: -
Unavailable: java python2 ts
Applications: -
Documentation: NO
Non-free algorithms: NO

Windows RT support: NO

GUI:
Win32 UI: YES
VTK support: NO

Media I/O:
ZLib: build (ver 1.2.11)
JPEG: build-libjpeg-turbo (ver 2.0.6-62)
WEBP: build (ver encoder: 0x020f)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.0.10)
JPEG 2000: build (ver 2.3.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES

Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.91.100)
avformat: YES (58.45.100)
avutil: YES (56.51.100)
swscale: YES (5.7.100)
avresample: YES (4.0.0)
GStreamer: NO
DirectShow: YES
Media Foundation: YES
DXVA: NO

Parallel framework: Concurrency

Trace: YES (with Intel ITT)

Other third-party libraries:
Intel IPP: 2020.0.0 Gold [2020.0.0]
at: C:/Users/appveyor/AppData/Local/Temp/1/pip-req-build-oduouqig/_skbuild/win-amd64-3.7/cmake-build/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2020.0.0)
at: C:/Users/appveyor/AppData/Local/Temp/1/pip-req-build-oduouqig/_skbuild/win-amd64-3.7/cmake-build/3rdparty/ippicv/ippicv_win/iw
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)

OpenCL: YES (NVD3D11)
Include path: C:/Users/appveyor/AppData/Local/Temp/1/pip-req-build-oduouqig/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load

Python 3:
Interpreter: C:/Python37-x64/python.exe (ver 3.7.5)
Libraries: C:/Python37-x64/libs/python37.lib (ver 3.7.5)
numpy: C:/Users/appveyor/AppData/Local/Temp/1/pip-build-env-b6d1tida/overlay/Lib/site-packages/numpy/core/include (ver 1.14.5)
install path: python

Python (for build): C:/Python27-x64/python.exe

Java:
ant: NO
JNI: C:/Program Files/Java/jdk1.8.0/include C:/Program Files/Java/jdk1.8.0/include/win32 C:/Program Files/Java/jdk1.8.0/include
Java wrappers: NO
Java tests: NO

Install to: C:/Users/appveyor/AppData/Local/Temp/1/pip-req-build-oduouqig/_skbuild/win-amd64-3.7/cmake-install

7

@kbarni, It seems many users are affected by this. I found the below open issue on Github. The bug seems to have been there for a long time. Given Bayer* layouts are widely used, anyone processing camera outputs directly is affected.

8

Forget your plt.imshow and show us matlab numerical result.
In python I can do this

import cv2 as cv
import numpy as np

# Simple matrix in BayerRG layout of a 6x12 image with all red pixels
img  = np.zeros(shape=(4,4),dtype=np.uint8)
img[0,:] = np.array([200,100, 200, 100])
img[1,:] = np.array([100,50, 100,50])
img[2,:] = np.array([20,10, 20, 10])
img[3,:] = np.array([10,5, 10,5])
img_BayerRG2RGB = cv.cvtColor(img, cv.COLOR_BayerRG2RGB)
print("Original image")
print( img)
print("BayerRG2RGB")
print(img_BayerRG2RGB)
img_BayerRG2BGR = cv.cvtColor(img, cv.COLOR_BayerRG2BGR)
print("BayerRG2BGR")
print(img_BayerRG2BGR)

numerical results
Original image

[[200 100 200 100]
 [100  50 100  50]
 [ 20  10  20  10]
 [ 10   5  10   5]]
BayerRG2RGB
[[[ 50  78 110]
  [ 50  78 110]
  [ 50 100 110]
  [ 50 100 110]]

 [[ 50  78 110]
  [ 50  78 110]
  [ 50 100 110]
  [ 50 100 110]]

 [[ 28  10  20]
  [ 28  10  20]
  [ 28  33  20]
  [ 28  33  20]]

 [[ 28  10  20]
  [ 28  10  20]
  [ 28  33  20]
  [ 28  33  20]]]
BayerRG2BGR
[[[110  78  50]
  [110  78  50]
  [110 100  50]
  [110 100  50]]

 [[110  78  50]
  [110  78  50]
  [110 100  50]
  [110 100  50]]

 [[ 20  10  28]
  [ 20  10  28]
  [ 20  33  28]
  [ 20  33  28]]

 [[ 20  10  28]
  [ 20  10  28]
  [ 20  33  28]
  [ 20  33  28]]]

I read this and I cannot find numerical error

1 Like
9

perhaps this is an issue of poor documentation.

documentation should precisely define what BayerRG, … mean in terms of the four possible color filter variants.

even that link you gave (Example: Bayer Filter Demosaic · codeplaysoftware/visioncpp Wiki · GitHub) uses words such as “even” and “odd”, which are completely ambiguous given the indexing in matlab and colloquially is 1-based, while it’s 0-based in regular programming languages.

this needs PRECISE documentation to be useful. that’s the first problem. everything else resolves itself upon precise documentation.

10

@laurent.berger, I won’t just “forget about” the plt.imshow - image display is where most users start. I also have no idea why you have introduced a more complicated example when my simpler example with all red can reproduce the issue. In my view, your example just creates more confusion. And, why ask for my version if you didn’t have the intention to reproduce my results yourself.

Now, for the sake of the community, here’s my MATLAB numerical result. I cannot just copy/paste output from the link I sent earlier, so I need to describe it. If I call the MATLAB function disp(J) in my example, I get this:
(:, :, 1), the red channel = 12x6 matrix of all 255s
(:, :, 2 and (:, :, 3)), the green and blue channels = 12x6 matrix of all 0s

Clearly, the MATLAB routine produces an all red image as expected.

I’m seeing the issue in your own example. The first pixel’s Bayer pattern in your original image is 200, 100, 100, and 50 (elements (1, 1), 1, 2), (2, 1), and (2, 2) of your matrix). This means R is the dominant color of this pixel, if the pattern is interpreted as BayerRG (aka RGGB). Yet, the first pixel in your BayerRG2RGB OpenCV result is [ 50 78 110], which is a blue-dominant pixel.

11

After additional extensive research, I have further narrowed down the source of all this confusion surrounding BayerRG → RGB conversion. And that source is this:

What the rest of world, including:

  1. MATLAB: see link in my previous message
  2. Camera manufacturer Basler: Bayer to RGB and missed pixel value at last row and line

consider as Bayer RG (aka RGGB), OpenCV considers as BayerBG (aka BGGR): OpenCV: Color conversions

Specifically, the rest of the world uses the top left 2x2 sub-matrix of the CFA (Color Filter Array) to name the Bayer pattern. OpenCV, on the other hand, uses the 2x2 sub-matrix that starts at the 2nd row and 2nd column of the CFA.

Does anyone know why OpenCV deviates from the rest of the world in this way in naming the Bayer patterns?

1 Like
12

probably no reason at all, except the one person who bothered to implement it, implemented it wrong, and nobody noticed.

I think the best thing would be to open this as an issue on the github and see how the core developers want to deal with it. they might be inclined to add alternative enum values that do it right (maintaining old ones for compatibility), or cover the defect with documentation.

or maybe one of them has better insight and can somehow flip the situation so it makes sense.

1 Like
13

New Github issue created: OpenCV's Bayer naming differs from the rest of the world, resulting in unexpected demosaicing results · Issue #19629 · opencv/opencv · GitHub

2 Likes