clustering is right. flann might be useful in this situation. I disagree with k-means, since there is no k here.
flann
k
maybe you want to use mean shift.