FileStorage Read fails > 32Gb. (Bug #4097)

Added by Kirk Mac about 10 years ago. Updated about 10 years ago.

Status:Cancelled Start date:2015-01-05
Priority:Normal Due date:
Assignee:Vadim Pisarevsky % Done:

0%
Category:core
Target version:2.4.11
Affected version:2.4.9 (latest release) Operating System:Linux
Difficulty: HW Platform:x64
Pull request:

Description

My computer has 320 GB of RAM, Ubuntu 12.04 x64, and OpenCV 2.4.9.1.

I have a yml file containing a 10643x100000 float matrix (saved by mexopencv, head and tail verified manually). The file is 4.4 GB. The Mat should be ~4 GB.
However, during the FileStorage read, I see the RAM usage go up around 32 GB, and then there's a segfault.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6d13598 in cvReadRawDataSlice () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4

Here's the backtrace from gdb:

#0  0x00007ffff6d13598 in cvReadRawDataSlice () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4
#1  0x00007ffff6d149c6 in cvReadRawData () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4
#2  0x00007ffff6d156b0 in icvReadMat(CvFileStorage*, CvFileNode*) () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4
#3  0x00007ffff6d1e9d3 in cvRead () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4
#4  0x00007ffff6d20a62 in cv::read(cv::FileNode const&, cv::Mat&, cv::Mat const&) () from /home/kam/ssd/kam/repos/opencv-install/lib/libopencv_core.so.2.4
#5  0x00000000004150a1 in operator>><cv::Mat> (value=..., n=...) at /ssd/kam/repos/opencv-install/include/opencv2/core/operations.hpp:3169

History

#1
Updated by Vadim Pisarevsky about 10 years ago

Hello!

thanks for the bug report. In fact, this is not a bug, this is shortcoming of the current implementation, where each single number takes the whole CvFileNode structure, which takes 24 bytes on a 64-bit machine. We are planning to revise the internals of CvFileStorage, where large arrays will be encoded using base64 scheme and will not be stored internally as a sequence of CvFileNode's, but this is time-consuming project. Right now the only workaround I see is to use some different mechanism to store matrices, e.g. right in binary form.
I know, there is some code floating around to load old-style mex files right into OpenCV cv::Mat.

Regards,
Vadim

  • Status changed from New to Cancelled

Also available in: Atom PDF