Persistance,cpp / FileStorage produces and assumes invalid XML (Bug #976)

oops, the formatting is broken. One more try:

#include "opencv2/opencv.hpp" 
#include <string>
#include <iostream>

using namespace cv;
using namespace std;

int main(int, char**)
{
    [[FileStorage]] fs("_a.xml", CV_STORAGE_WRITE);
    cvWriteComment(fs.fs, "комментарий по-русски", false);
    fs << "a" << string("текст");
    fs << "b" << string("еще немного текста");
    fs.release();
    fs.open("_a.xml", CV_STORAGE_READ);
    string s = fs[[a]];
    string s1 = fs[[b]];
    cout << "read a: " << s << " and b: " << s1 << endl;
    return 0;
}

of course, the sample will work if the source code has UTF-8 encoding, or if you converted the strings and comments to UTF-8 before passing them to OpenCV.

Ok, I just noticed one thing, while looking at the encodings in Notepad++ again: If you use UTF-8 encoding on a file that doesn't contain any extended characters, and you leave out the BOM, the encodings UTF-8 and ANSI are indistinguishable. So you convert something with Notepad++ to UTF-8, reload it, and it still shows up as plain ANSI.

Still, if somebody puts in some special character, then suddenly it all makes some difference (doesn't have to be some fancy Asian unicode stuff, I think a french letter like é or a § sign would be enough).

And another thing: Since UTF-8 encoding is assumed when there is no encoding-tag anyway, putting "UTF-8" or no tag at all should make absolutely no difference.

ASCII/ANSI is a subset of UTF-8, so any UTF-8 document containing only characters with codes 32-127 is also valid ASCII/ANSI document.

From your notes about the old code running fine and some strange garbage symbols appearing in the printed text strings and the file name I conclude that there is something wrong in your code, or the file, containing the source code. I suggest you to check the source code with a hex editor and make sure it's not UTF-16 file and it does not contain any strange characters. It the code, producing the attached XMLs, is fine, can you supply the code as well?

Replying to [comment:9 vp153]:

From your notes about the old code running fine and some strange garbage symbols appearing in the printed text strings and the file name I conclude that there is something wrong in your code, or the file, containing the source code. I suggest you to check the source code with a hex editor and make sure it's not UTF-16 file and it does not contain any strange characters.

I did that before (even checked the Visual Studio solution with an editor). Anyway, I put everything into a clean new project, and now your example works correctly with the latest version.

#include "opencv2/opencv.hpp" 
#include <string>
#include <iostream>

using namespace cv;
using namespace std;

int main()
{
    [[FileStorage]] fs("_a.xml", CV_STORAGE_WRITE);
    cvWriteComment(fs.fs, "aaaaa", false);
    //fs << "a" << string("bbbbääöö");
    fs << "a" << string("bbbbb");
    fs << "b" << string("ccccc");
    fs.release();
    fs.open("_a.xml", CV_STORAGE_READ);
    string s = fs[[a]];
    string s1 = fs[[b]];
    cout << "read a: " << s << " and b: " << s1 << endl;
    return 0;
}

However, if I comment out the line with the extended ASCII characters, I get this error:

        _ASSERTE((unsigned)(c + 1) <= 256);

It's caused by OpenCV here, persistence.cpp, line 1887:

                    if( !isalnum(c) )

Best regards, Linus

ok, in the latest persistence.cpp I replaced all the is<> with custom functions that should not throw exceptions. Hopefully, it will now work well.

Thanks again for your fast fix. Indeed, now the example code runs fine, even with special characters! However, Notepad++ shows the produced XML file like this:

<?xml version="1.0" encoding="UTF-8"?>
<opencv_storage>
<!-- aaaaa -->
<a>"bbbb奶��a>
<b>ccccc</b>
</opencv_storage>

In the "Encoding" menu, it shows "UTF-8 (without BOM)". It probably detected this because of the UTF-8 encoding tag. When I switch to ANSI, it all looks ok.

It also looks ok when I manually edit the encoding-tag to "ASCII". Apparently, then Notepad++ chooses the right encoding.:

<?xml version="1.0" encoding="ASCII"?>
<opencv_storage>
<!-- aaaaa -->
<a>"bbbbääöö"</a>
<b>ccccc</b>
</opencv_storage>

• What encoding is specified, i.e. what is determined by the encoding tag, by the BOM, or the default option.
• The actual underlying character encoding of the string data.

Until now, OpenCV always puts ASCII/ANSI. That's why I suggested "ok, let's put the ASCII tag into the XML file".

The good thing for me personally is, as I realized and you pointed out: If you stick to "standard" US 32 - 127 chars, UTF-8 and ASCII/ANSI are indistinguishable...

• OpenCV encodes the actual file content to UTF-8
• or the encoding tag says ASCII

I'm mentioning this, because in European languages extended chars ( 128 - 255) are very common, like é, ö, etc.

utf-8 is the encoding we should use. Other alternatives are inferior. It's developer who should care that the written text strings have UTF-8 encoding. OpenCV takes "const char*" or "std::string", so it knows nothing about the currently used encoding. It's not in a scope of computer vision library to handle different encodings, so we just assume utf-8. On Linux, MacOSX and virtually all other Unix systems UTF-8 is the default encoding, so everything should work out-of-the-box. On Windows some software sometimes uses UTF-16, or iso-8859-1 or another single-byte "local" encoding. In this case the written strings should first be converted to UTF-8. I will add this note to the reference manual.

I suppose, the ticket can be closed, right?

Oh ok, I get it now. As long as it all works when you pass an actual UTF-8 string, that's fine. (And as long as the documentation says: Use UTF-8 with persistence!).

I understand your point about OpenCV, and I agree. I won't do any further tests, as I'm not using UTF-8 anyway. Right now I'm satisfied, thanks for the heads up and discussion.

Can be closed, thanks again

ok, thank you too! :)
one more question before closing the ticket. If no encoding is specified

<?xml version="1.0"?>
<opencv_storage>
<!-- aaaaa -->
<a>"bbbbääöö"</a>
<b>ccccc</b>
</opencv_storage>

is the file displayed correctly in Notepad++?
I just checked and it works perfectly well on mac. If it improves situation on Windows, I can remove encoding specification from XML.

Regards,
Vadim

Vadim, please excuse my delay.

Replying to [comment:15 vp153]:

one more question before closing the ticket. If no encoding is specified
[...]
is the file displayed correctly in Notepad++?

Yes, it is displayed correctly then. But it's just because of the automatic "encoding recognition" by Notepad++ I guess, which tries to read the encoding-tag.

I just checked and it works perfectly well on mac. If it improves
situation on Windows, I can remove encoding specification from XML.

Whether or not it improves the situation should depend on the app reading the XML file. According to standard, there should be no difference (since the UTF-8 specifier could also be omitted). It seems in the case where it's ommited, Notepad++ did some heuristics on character frequency to decide how to display the text to the user. However, I personally wouldn't say Notepad++ is the "target app" that needs to be supported. It probably just tries to display the text to the user best -- so it might do things differently then a certain XML parser app would do.

Replying to [comment:16 vp153]:

some more changes.

Great, the r5253 patch is probably the most sensible thing to do. It makes everything more flexible and let's the user take care (makes sense, as the user also has to worry about the encoding). I'll personally use the "ASCII" tag for my case now.

Thanks for fixing!

Linus