CvMLData::read_csv cannot load large data file (Bug #510)

Added by Fei DUAN over 14 years ago. Updated almost 13 years ago.

Status:Done Start date:
Priority:High Due date:
Assignee:Vadim Pisarevsky % Done:

0%
Category:ml
Target version:-
Affected version: Operating System:
Difficulty: HW Platform:
Pull request:

Description

when the dimension of the data are above several thousand, CvMLData::read_csv() method always fail.

My Testing platform is Windows XP / Win7

History

#1
Updated by Victor Eruhimov over 14 years ago

What is the number of rows? Can't it be that the data just doesn't fit into operating memory?

  • Status deleted (Open)

#2
Updated by Fei DUAN over 14 years ago

Replying to [comment:1 relrotciv]:

What is the number of rows? Can't it be that the data just doesn't fit into operating memory?

No, the data does can fit into the RAM. I wrote my own loading function, there was no load failures. Actually, the number of rows is less than 5,000, but the columns of each row is about 2,000. Maybe you can generate a random array and try CvMLData;;read_csv() ?

#3
Updated by Vadim Pisarevsky over 14 years ago

fixed in r4275

  • Status set to Done
  • (deleted custom field) set to fixed

#4
Updated by Andrey Kamaev almost 13 years ago

  • Category set to ml
  • Assignee changed from Maria Dimashova to Vadim Pisarevsky

Also available in: Atom PDF