How to debayer? (perform convolution with a step size of 2)

C++
1

I want to perform a convolution operation with a step size of 2, but I found that this param is not exist in cv::filter2D

2

May be because it’s not a convolution.

Use filter2d and extract only result you need using a loop

3

I found a backdoor using tensorflow model

import numpy as np
import tensorflow as tf
import tensorflow.python.framework.convert_to_constants as tfpy

import cv2 as cv
# for frozen graph https://medium.com/@sebastingarcaacosta/how-to-export-a-tensorflow-2-x-keras-model-to-a-frozen-and-optimized-graph-39740846d9eb

model1 = tf.keras.Sequential(
    [
        tf.keras.layers.Conv2D(1,kernel_size=(2,2), strides=(2,2), activation=None, use_bias=False, name="layer1", kernel_initializer='one')
    ]
)
x = tf.ones((1,9, 9, 1))
y = model1(x)
h2d = np.zeros(shape=(2, 2, 1,1), dtype=np.float32)
h2d[0:2, 0:2, 0, 0] = [[0.2, 0.4],[0.2, 0.2]]
model1.layers[0].set_weights([h2d])


full_model = tf.function(lambda x: model1(x))
full_model = full_model.get_concrete_function(
    tf.TensorSpec(model1.inputs[0].shape, model1.inputs[0].dtype))
frozen_func = tfpy.convert_variables_to_constants_v2(full_model)
tf.io.write_graph(graph_or_graph_def=frozen_func.graph,
                  logdir="",
                  name="frozen_graph.pb",
                  as_text=False)

model_ocv = cv.dnn.readNetFromTensorflow(r"frozen_graph.pb")
img = cv.imread(cv.samples.findFile("lena.jpg"))
blob_img = cv.dnn.blobFromImage(img[:,:,0])
model_ocv.setInput(blob_img)
out = model_ocv.forward()
cv.imshow("conv",out[0, 0,:,:].astype(np.uint8))
cv.waitKey()
4

Unfortunately, I’m using C++

5

May be you can show us your code before.

For dnn solution translate in C++ is straitghforward

model_ocv = cv.dnn.readNetFromTensorflow(r"frozen_graph.pb")
img = cv.imread(cv.samples.findFile("lena.jpg"))
blob_img = cv.dnn.blobFromImage(img[:,:,0])
model_ocv.setInput(blob_img)
out = model_ocv.forward()
cv.imshow("conv",out[0, 0,:,:].astype(np.uint8))
cv.waitKey()

For tensorflow code

import numpy as np
import tensorflow as tf
import tensorflow.python.framework.convert_to_constants as tfpy

import cv2 as cv
# for frozen graph https://medium.com/@sebastingarcaacosta/how-to-export-a-tensorflow-2-x-keras-model-to-a-frozen-and-optimized-graph-39740846d9eb

model1 = tf.keras.Sequential(
    [
        tf.keras.layers.Conv2D(1,kernel_size=(2,2), strides=(2,2), activation=None, use_bias=False, name="layer1", kernel_initializer='one')
    ]
)
x = tf.ones((1,9, 9, 1))
y = model1(x)
h2d = np.zeros(shape=(2, 2, 1,1), dtype=np.float32)
h2d[0:2, 0:2, 0, 0] = [[0.2, 0.4],[0.2, 0.2]]
model1.layers[0].set_weights([h2d])


full_model = tf.function(lambda x: model1(x))
full_model = full_model.get_concrete_function(
    tf.TensorSpec(model1.inputs[0].shape, model1.inputs[0].dtype))
frozen_func = tfpy.convert_variables_to_constants_v2(full_model)
tf.io.write_graph(graph_or_graph_def=frozen_func.graph,
                  logdir="",
                  name="frozen_graph.pb",
                  as_text=False)

You can run in python and I think it should be possible to set filter values in C++ using layer(1).blob

6

I currently do not have a complete code. I now have an image in BayerRG format, which is an 8UC1 image. The structure is:
R G R G
G B G B
R G R G
G B G B
“I want to design a convolution kernel myself, suppress other colors (including white), and extract only red. I expect a 2 * 2 convolution kernel with a positive red position and a negative blue green position, but I only need to extract at the red point. Therefore, I hope to obtain a convolution result that is twice as small as the original image by setting the step size to 2, which can achieve maximum efficiency.”.
Or, do you know any other better way to quickly extract red and blue colors on BayerRG?
thank.

7

What’s about cvtColor and color spaces?

8

I can use cvtColor and BayerRG2HSV to implement it, but they calculate too much unnecessary data, so I want to find a faster method (our device is really limited in computational complexity).

9

Perhaps I am missing something here, but why not build a mask and use that to pluck out the specific pixels you need?

1 Like
10

“convolution” is the wrong word here, total misnomer, you’re not convolving neighbour pixels into a single result

IF it is really so, that you only want to extract the red bytes, some “drop every 2nd row/col” schema might work (like hinted by @Phyzx (just maps, not masks)), which could be implemented quite fast using remap :

# dummy input, we want to extract the '1' s from it
a = np.array([[1,2,1,3,1,2,1,3],
              [2,3,2,3,2,3,2,3],
              [1,2,1,3,1,2,1,3],
              [2,3,2,3,2,3,2,3],
              [1,2,1,3,1,2,1,3],
              [2,3,2,3,2,3,2,3]], dtype=np.uint8)

print(a.dtype,a.shape)

# mapping with new pixel positions
# we drop every 2nd row/col here:
m1 = np.array([[[0,0],[2,0],[4,0],[6,0]],
               [[0,2],[2,2],[4,2],[6,2]],
               [[0,4],[2,4],[4,4],[6,4]]],dtype=np.float32)
z = cv.remap(a,m1,None,cv.INTER_LINEAR)
print(z.dtype,z.shape)
print(z)

---------------------------
# out
uint8 (6, 8)
uint8 (3, 4)
[[1 1 1 1]
 [1 1 1 1]
 [1 1 1 1]]

todo (ofc. !) : find a proper formula to generate the mapping :slight_smile: