BOWImgDescriptorExtractor and Mat.type() == 16 issue (Bug #1879)

#Bugs related to commit:
r8280 | vp153 | 2012-04-30 07:33:52 -0700 (Mon, 30 Apr 2012)

#invoking call from
Revision 8280 trunk/opencv/modules/features2d/include/opencv2/features2d/features2d.hpp

#Symptoms started to occur in this area during the last version change:
Revision 8280 trunk/opencv/modules/core/src/stat.cpp
Revision 8045 trunk/opencv/modules/core/src/stat.cpp

#Reverting to prior revision to confirm issue in version 8280 number:
cd ~/SDK/opencv/trunk/
svn up --revision 8279

#Rebuilt... and confirmed issue with type disappears...

"It is difficult to build a brick house on shifting sands" =)
Cheers,
J

Hi Joel,

thank you for the report. Could you attach the sample code and data (vocabulary matrix and imgMat) to reproduce your problem? Basing on your messages it's hard to find the reason of the problem. From the issue topic and first message it follows that you suppose that the problem is in imgMat type (16, i.e. CV_8UC3). But assert() in batchDistance checks the things which don't depend on imgMat type:

1. type = = src2.type() - types of vocabulary descriptors and descriptors computed in imgMat (using DescriptorExtractor which you passed to BOWImgDescriptorExtractor constructor) must be equal;
2. src1.cols = = src2.cols - dimensions of descriptors computed in imgMat and descriptors in vocabulary matrix must be equal;
3. type = = CV_32F || type = = CV_8U - descriptor elements must be float or uchar;

Please, be sure that you use the same type of DescriptorExtractor to build vocabulary and to compute image descriptor (using BOWImgDescriptorExtractor) and descriptors type is CV_32F or CV_8U.

Hi Joel,

Thanks for the code. Unfortunately it can't help without the data. I have to suppose even a type of the descriptor extractor you used, have to build some vocabulary, etc. Maybe the problem is specific for the particular descriptor extractor type and your data.

So I can just assume the reason of the problem from the fact that it happens when you switch to r8280 (leaving the rest without changes). As you got an assertion fail in the line stat.cpp:1770, one of the following conditions must be false: type = = src2.type(), src1.cols = = src2.cols, type = = CV_32F || type = = CV_8U. I didn't understand exactly which one was false from your messages, sorry. But I have an assumption:

• (?) You trained your vocabulary using <r8280.
• (?) You use some Opponent descriptor extractor.
• (!) There was a bug in DescriptorExtractor::create in <r8280 for Opponent descriptors. It created not Opponent descriptor, eg SURF instead of OpponentSURF. And it was fixed in r8280 http://code.opencv.org/projects/opencv/repository/revisions/8280/diff/trunk/opencv/modules/features2d/src/descriptors.cpp
• (?) So for the r8280 you have src1.cols ! = src2.cols, specifically src1.cols = = 3 * src2.cols, because now you have really Opponent descriptor which is 3 time longer than the pure one.

Please, check this assumption. If I'm wrong, could you say me which of the three conditions is false? It would be also very useful if you give me the vocabulary, imgMat and say me the descriptor extractor type you used.

Thanks Maria,
I ran several training permutations, and the associated regression tests for precision/recall with:

... HARRIS SURF FlannBased
... PyramidHARRIS SURF BruteForce
... SURF SURF BruteForce
... SURF OpponentSURF BruteForce
... SURF OpponentSURF FlannBased
... DynamicSURF OpponentSURF FlannBased

Although, your observations would explain the odd plot data for the last 3 tests that were run. I was expanding the vocabulary limits by falsely assuming the odd results may have had something to do with high Hessian thresholds truncating the vocabulary sample points (8000+ images).

Currently I'm still porting my code to Ubuntu 12.04 LTS, but I will do an svn up and re-run the vocabulary regression testing suites over the weekend. Notably, libtbb-dev is now a deb package... =)

These are the initial data sets I used:
http://www.vision.caltech.edu/Image_Datasets/Caltech101/#Download
http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz

Cheers,
J

Current svn lib is not functional even with OpenCV example code.
All permutations will fail... =(

It appears to currently throw an error here:
trunk/opencv/modules/features2d/src/descriptors.cpp

OpponentColorDescriptorExtractor::OpponentColorDescriptorExtractor( const Ptr<DescriptorExtractor>& _descriptorExtractor ) :
        descriptorExtractor(_descriptorExtractor)
{
    CV_Assert( !descriptorExtractor.empty() );
}

#testing data with OpenCV svn head:
cd ~/SDK/opencv/trunk/opencv/samples/cpp
./bagofwords_classification VOC2010 test SURF OpponentSURF BruteForce

OpenCV Error: Assertion failed (!descriptorExtractor.empty()) in OpponentColorDescriptorExtractor, file /home/yourmom/SDK/opencv/trunk/opencv/modules/features2d/src/descriptors.cpp, line 117
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/yourmom/SDK/opencv/trunk/opencv/modules/features2d/src/descriptors.cpp:117: error: (-215) !descriptorExtractor.empty() in function OpponentColorDescriptorExtractor

Aborted (core dumped)

#testing data with auto loading script:

OpenCV Error: Assertion failed (!descriptorExtractor.empty()) in OpponentColorDescriptorExtractor, file /home/yourmom/SDK/opencv/trunk/opencv/modules/features2d/src/descriptors.cpp, line 117
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/yourmom/SDK/opencv/trunk/opencv/modules/features2d/src/descriptors.cpp:117: error: (-215) !descriptorExtractor.empty() in function OpponentColorDescriptorExtractor

Aborted (core dumped)

 
-- General configuration for OpenCV 2.4.0 =====================================
-- Version control:                 svn:8386
-- 
--   Platform:
--     Host:                        Linux 3.2.0-20-generic x86_64
--     CMake:                       2.8.7
--     CMake generator:             Unix Makefiles
--     CMake build tool:            /usr/bin/make
--     Configuration:               RELEASE
-- 
--   C/C++:
--     Built as dynamic libs?:      YES
--     C++ Compiler:                /usr/bin/c++ (ver 4.6.3)
--     C++ flags (Release):         -Wall -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -ffunction-sections -O3 -DNDEBUG   -DNDEBUG
--     C++ flags (Debug):           -Wall -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -ffunction-sections -g  -O0 -DDEBUG -D_DEBUG -ggdb3
--     C Compiler:                  /usr/bin/gcc
--     C flags (Release):           -Wall -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -ffunction-sections -O3 -DNDEBUG  -DNDEBUG
--     C flags (Debug):             -Wall -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -ffunction-sections -g  -O0 -DDEBUG -D_DEBUG -ggdb3
--     Linker flags (Release):      
--     Linker flags (Debug):        
-- 
--   OpenCV modules:
--     To be built:                 calib3d contrib core features2d flann gpu highgui imgproc legacy ml nonfree objdetect photo python stitching ts video videostab
--     Disabled by user:            world
--     Disabled by dependency:      -
--     Unavailable:                 androidcamera java
-- 
--   GUI: 
--     QT 4.x:                      YES (ver 4.8.1 EDITION = OpenSource)
--     QT OpenGL support:           YES (/usr/lib/x86_64-linux-gnu/libQtOpenGL.so)
--     OpenGL support:              YES (/usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so /usr/lib/x86_64-linux-gnu/libSM.so /usr/lib/x86_64-linux-gnu/libICE.so /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libXext.so)
-- 
--   Media I/O: 
--     ZLib:                        /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.3.4)
--     JPEG:                        /usr/lib/x86_64-linux-gnu/libjpeg.so (ver )
--     PNG:                         /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.2.46)
--     TIFF:                        /usr/lib/x86_64-linux-gnu/libtiff.so (ver 42)
--     JPEG 2000:                   /usr/lib/x86_64-linux-gnu/libjasper.so (ver 1.900.1)
--     OpenEXR:                     /usr/lib/libImath.so /usr/lib/libIlmImf.so /usr/lib/libIex.so /usr/lib/libHalf.so /usr/lib/libIlmThread.so (ver 1.6.1)
--     OpenNI:                      YES (ver 1.5.2, build 23)
--     OpenNI PrimeSensor Modules:  YES (/usr/lib/libXnCore.so)
-- 
--   Video I/O:
--     DC1394 1.x:                  NO
--     DC1394 2.x:                  YES (ver 2.2.0)
--     GStreamer:                   
--       base:                      YES (ver 0.10.36)
--       app:                       YES (ver 0.10.36)
--       video:                     YES (ver 0.10.36)
--     UniCap:                      YES (ver 0.9.5)
--     UniCap ucil:                 YES (ver 0.9.5)
--     PvAPI:                       NO
--     V4L/V4L2:                    Using libv4l (ver 0.8.6)
--     Xine:                        YES (ver 1.2.0)
--     FFMPEG:                      YES
--       codec:                     YES (ver 54.17.101)
--       format:                    YES (ver 54.3.100)
--       util:                      YES (ver 51.49.100)
--       swscale:                   YES (ver 2.1.100)
--       gentoo-style:              YES
-- 
--   Other third-party libraries:
--     Use IPP:                     NO
--     Use TBB:                     YES (ver 4.0 interface 6000)
--     Use Cuda:                    NO
--     Use Eigen:                   YES (ver 2.0.17)
--     Use Clp:                     NO
-- 
--   Python:
--     Interpreter:                 /usr/bin/python (ver 2.7.3)
--     Libraries:                   /usr/lib/libpython2.7.so
--     numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.6.1)
--     packages path:               lib/python2.7/dist-packages
-- 
--   Documentation:
--     Build Documentation:         YES
--     Sphinx:                      /usr/bin/sphinx-build (ver 1.1.3)
--     PdfLaTeX compiler:           /usr/bin/pdflatex
-- 
--   Tests and samples:
--     Tests:                       YES
--     Performance tests:           YES
--     Examples:                    YES
-- 
--   Install path:                  /usr/local/opencv
-- 
--   cvconfig.h is in:              /home/yourmom/SDK/opencv/trunk/opencv/release
-- -----------------------------------------------------------------
-- 
-- Configuring done
-- Generating done 

The problem in creation of surf descriptor extractor in bagofwords_classification sample was due to SURF, SIFT are moved to nonfree module in 2.4 and inherited from cv::Algorithm. Now to use SURF and SIFT it is needed to include nonfree module and add cv::initModule_nonfree() (see doc for cv::Algorithm::create). I did this for the sample in r8398. Please, check that it works for you now.

Joel, how are things with the assertion in stat.cpp, line 1770? Do you still have the fail there?

Maria Dimashova wrote:

The problem in creation of surf descriptor extractor in bagofwords_classification sample was due to SURF, SIFT are moved to nonfree module in 2.4 and inherited from cv::Algorithm. Now to use SURF and SIFT it is needed to include nonfree module and add cv::initModule_nonfree() (see doc for cv::Algorithm::create). I did this for the sample in r8398. Please, check that it works for you now.

Joel, how are things with the assertion in stat.cpp, line 1770? Do you still have the fail there?

Maybe cv::initModule_nonfree() should be mentioned at the documetation of SIFT/SURF/features2d.

@Guanta Alecho
Indeed, this it is a new library feature that will likely impact other examples.

@Maria Dimashova
There seems to be a 2.5GB Memory size limitation for r8429 vocabulary, and currently ignores memoryUse() setting in VocabTrainParams(). Additionally, the training program will crash if limit is set to 12000.

12GB bug:

...
Extracting VOC data...
Computing descriptors...
bagofwords_classification: /home/yourmom/bagofwords_classification.cpp:2201: cv::Mat trainVocabulary(const string&, VocData&, const VocabTrainParams&, const cv::Ptr<cv::FeatureDetector>&, const cv::Ptr<cv::DescriptorExtractor>&): Assertion `bowTrainer.descripotorsCount() == maxDescCount' failed.

2.5GB limit bug:

----------------------------------------------------------------
detectorType: DynamicSURF
descriptorType: OpponentSURF
matcherType: FlannBased

trainObjClass: cat
vocabSize: 777
memoryUse: 9000
descProportion: 0.35

descPercent: 0.03
targetRatio: 0.61
balanceClasses: 1
----------------------------------------------------------------

Reading vocabulary... 
Extracting VOC data...
Computing descriptors...
Breaking due to full memory ( descriptors count = 551594; descriptor size in bytes = 1536; all used memory = 847248384
Maximum allowed descriptor count: 551594, Actual descriptor count: 551594
Training vocabulary...

I will let the tests permute the classifier options over the next few days... The vocabulary limit will not span all of the imported balanced item classes (about 25% of 8000+ images), but these tests should determine if the opencv BoW lib is operational.

Thanks again,
J

The memory allocation limits how much data I can test, but early results are showing decent precision versus recall plots for traditionally difficult items.

The first test completed successfully with 3400 images :
... DynamicSURF OpponentSURF FlannBased

Notably, the old balanced precision/recall curve bug for the primary class appears to no longer be incorrectly fixed at a single value. However, the test scripts will audit for the old 2.3.1 anomaly over the weekend.

I will post the other test configuration results on Monday, and will attempt to investigate the memory allocation issues.

Thanks again Maria, and please let me know how I can support your work...
J
:-)

Precision/recall performance for BoW configurations successfully tested with 3400 image set:
... DynamicSURF OpponentSURF Bruteforce
... DynamicSURF OpponentSURF FlannBased
... PyramidHARRIS OpponentSURF FlannBased
... SURF OpponentSURF Bruteforce

I will continue to run tests with larger data sets on r8481, as the 64bit format resident memory limit manifests as an incorrect size estimate in some examples. Upon loading around ~6400 images for this set only %60 of available resident memory is correctly allocated, and thus will likely limit the upper bound on the descriptor counts in other examples.

Thanks again everyone, :-)
J