Efficiently loading datasets in opencv C++

C++
,
1

So i am looking into training a machine learning model for hand written text recognition using OpenCV and C++. I would like to clarify that i am not looking for a deep learning approach.
Now,obviously you need a dataset,and there should be a way to load this dataset so that you can then extract features from that dataset,and then later on train it.
One way to do this is using the following syntax

vector<string> fn
cv::glob(path,fn,true)
for(size_t k=0;k<fn.size();k++){
cv::Mat im=cv::imread(fn[i]);
//do other stuff

But will this method be optimal if the dataset contains a large number of images.
For example if I was to use this to read GTI vehicle dataset which has 6000 images,won’t this be inefficent.
Also,in python you can use sklearn’s train test and split method to divide your dataset into train,validation and testing.Assuming that I have extracted the features,how exactly can we split our dataset into train,test and validation in Opencv using C++.
So to summarize ;

  1. What are the methods to efficiently load large datasets so that we can later we can extract features ,and appropriately train a machine learnign model on it.
    2.How does train,test and validation split work in OpenCV C++.
2

so, what are you trying to use instead ?

there’s a TrainData class, there are loaders/parsers for special datasets , but none of them might be feasible for you

3

I was thinking of using conventional approaches like SVM/KNN.One such methodology is here

and svm+hog example is already mentioned in opencv samples
I also saw your post on Sequence files in OpenCV C++ - Stack Overflow
the module dataset in opencv-contrib tends to answer my questions on loading datasets and annotations.
But I would like to still know how do we go about doing the train-test-val split in C++.Is there a specific class for it.

4

you said that none of the parsers will be feasible,since I am working on IAM handwriting dataset what could be the problem.If these parsers won’t work should I stick to simply using the cv::glob method as mentioned in post.

5

so, it’s this ?

just curious, how do you plan to extract single chars from it (do you, even) ?
(it seems to have “whole word” images)

you’d also have to parse xml meta data to get to the ground truth responses, i guess.

6

yep that’s the one.
For starters most HWR would have a word segmentation step.once individual words are extracted,then you can go about training your ML model on it.
here is a link but its in python

as for parsing the xml meta data to get the ground truth,any tips on how to go about that.
Also if you can give pointers on how to use the trainData class for reading an image dataset that would be nice.I am not coming up with anything

7

it does not do that. you’re expected to create it with a single data Mat (each feature on a row) and a labels Mat (one for each feature/row)
where and how you get that data, is still up to you
however, there are some samples you could look at

8

hmmm,well in this case the features are already extracted.So it’s easy to use fstream based options to read it.
I mean for some time let’s not talk about handwritten recognition.
Let’s say i am talking about a general image classification problem like vehicle detection problem.If the dataset is say LFW or GTI,how do I proceed reading those images.

9

I think this example suits me much better.Could you explain the reasoning behind convert_to_ml function here.

10

as said before, opencv’s ml classes expect all train data in a single Mat,
this function copies HOG descriptors to rows in that Mat.
(later, an SVM is trained on that)

11

if you’re asking about how to most efficiently load data… don’t worry about that until you have measured the time it takes, and it’s a significant portion of the whole workflow.