Extract Graph Data from image

[stevemoretz]

I’m pretty new to both python and openCV. I just need it for one project. Users take picture of ECG with their phones and send it to the server I need to extract the graph data and that’s all.

Here’s a sample image :

Original Image Sample

I should first crop the image to have only the graph I think.As I couldn’t find a way I did it manually.

Here’s some code which tries to isolate the graph by making the lines white (It works on the cropped image) Still leaves some nasty noises and inacurate polygons at the end some parts are not detected :

import cv2
import numpy as np

img = cv2.imread('image.jpg')

kernel = np.ones((6,6),np.uint8)
dilation = cv2.dilate(img,kernel,iterations = 1)

gray = cv2.cvtColor(dilation, cv2.COLOR_BGR2GRAY)
#
ret,gray = cv2.threshold(gray,160,255,0)
gray2 = gray.copy()
mask = np.zeros(gray.shape,np.uint8)

contours, hier = cv2.findContours(gray,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
for cnt in contours:
    if cv2.contourArea(cnt) > 400:
        approx = cv2.approxPolyDP(cnt,
                                  0.005 * cv2.arcLength(cnt, True), True)
        if(len(approx) >= 5):
            cv2.drawContours(img, [approx], 0, (0, 0, 255), 5)
res = cv2.bitwise_and(gray2,gray2,mask)
cv2.imwrite('output.png',img)

Now I need to make it better. I found most of the code from different places and attached them togheter.

np.ones((6,6),np.uint8)

For example here if I use anything other than 6,6 I’m in trouble
also

cv2.threshold(gray,160,255,0)

I found 160 255 by tweaking and all other hardcoded values in my code what if the lighting on another picture is different and these values won’t work anymore?

And other than this I don’t still get the result I want some polygons are attached by two different lines from bottom and top!

I just want one line to go from beggining to the end.

Please guide me to tweak and fix it for more general use.

[stevemoretz]

import numpy as np
import cv2
# load image
img = cv2.imread("1.jpg")

# Convert BGR to HSV
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

# define range of black color in HSV
lower_val = np.array([0,0,0])
upper_val = np.array([50,100,105])

# Threshold the HSV image to get only black colors
mask = cv2.inRange(hsv, lower_val, upper_val)

# invert mask to get black symbols on white background
mask_inv = cv2.bitwise_not(mask)


cv2.imwrite('output.png',mask_inv);

Here’s another approach get’s better results but still depending on lighting it fails for some parts sometimes.Also sometimes it shows the grid

[crackwitz]

this isn’t trivial.

all the variation you get from random users using random smartphones taking random photos of random ECG plots…

some thoughts…

• rectify the snapshot so the plot becomes a real rectangular frontal view
  – look for “opencv document scanner”, replicating a “camera document scanner” is reasonably simple, unless your picture doesn’t lend itself to the simple approaches
• remove grid lines/dots
  – could be done with a fourier transform, masking the right high frequency components, and transforming back. or maybe not. depends on what you have to deal with.
• depending on picture quality, remove vignette and other coarse shading effects
• maybe threshold, can’t say without dealing with a bunch of examples
• don’t use findcontours, that’s drawing a line around blobs, but your blobs are the lines themselves.
• split the picture into its subplots, treat each individually
• I’d walk pixel column by pixel column, get centroid/mean/mode/median/… and that shall be the y-coordinate at that pixel column

[crackwitz]

here’s the thing unwarped, by eye.

FFT, zeroing out some hot spots corresponding to the grid:

inverse FFT:

some not-quite-thresholding only to show what’s there, not a vital image operation:

I used imagej for the FFT stuff because it was quick and handy. I used a random photo editor for the perspective transform and the contrast/“thresholding”

[stevemoretz]

Hi Thanks for all the information that you provided.
As I’m really new to this trying all these solutions would take a long time for me.
I’m not even a python developer, I do more javascript stuff application programming…

So it would be really helpful if you provide some example codes which would help to run and understand in a faster way.

Here are some more examples for now I cropped them manually.
Using the camera document scanner sounds like a great idea, I even tried it today but failed due to some problems, Maybe I should ask a separate question for that.

remove grid lines/dots

Tried a lot of examples even from opencv docs.The one that removes the lines from musical notes, couldn’t really get it working.Would love some sample code please.

nice tip about the findcontours.

splitting sounds like a great idea as it’s 12 plots on the same paper.

I’d like to walk column by pixel column get the black pixels and therefore coordinates but for now I have a lot of noise using the last code I provided.

So in ideal If I get only 3 Ys for each x I can get all the curve information.
But now I sure can’t get only 3 because there are thousands of little black points all over the place and some parts don’t have any points after using inRange hsv.

I would love some sample code please.
This is driving me crazy.

[crackwitz]

(I’ve added a few more pictures to my last reply)

this is not a weekend project.

there are many aspects of image processing and computer vision you would need to study.

the most important is that snapshots are BAD data and you need to literally whip your users because they won’t learn how to take decent photos otherwise.

you’ll notice that those pictures vary in sharpness, they have motion blur, the level of noise tells me they were taken under insufficient lighting, …

split the problem into steps. work on automating just one step at a time, and jump over the other steps using photoshop skills/fake/synthetic data.

I can’t give you code. that doesn’t help you. I can tell you the methods that might be worth a try but that isn’t a solution. there is a LOT more to a solution than just the general sketch of a solution. details need to be solved, which I’m just glancing over here.

let me repeat the steps:

• rectify the picture, which uses a perspective transformation/warp, but getting the corners is the real problem (or whip the user into taking a great upright frontal picture)
• remove the grid, which can use fourier transforms and blanking out frequency components, or it can be done some other way
• analyze the “clean” picture (idea as above) to extract an idealized description of an ECG curve from the pixels

[crackwitz]

related:

Extract Graph Data from image using the openCV?

Hello, I’m pretty new to both python and openCV. I just need it for one project. Users take picture of ECG with their phones and send it to the server I need to extract the graph data and that’s all. Here’s some code which tries to isolate the graph by making the lines white (It works on the cropped image) Still leaves some nasty noises and inacurate polygons at the end some parts are not detected : import cv2 import numpy as np img = cv2.imread('image.jpg') kernel = np.ones((6,6),np.uint8)…