Serialization - Implementation Notes

###

# Serialization

## Implementation Notes

Partial Function Template Ordering
Character Encoding
Template Invocation syntax
Partial Template Specialization
Specific Compiler/Library Issues

GCC 3.X
GCC 2.95
Intel 8.0
Visual C++ 7.1
Visual C++ 7.0
Visual C++ 6.0
Borland 5.64
Borland 5.51 and earlier
Dinkumware Library

### Partial Function Template Ordering

Not all C++ compilers correctly support partial function template ordering (PFTO). For these compilers, the following code will fail to compile:

    template<class Archive, class T>
    void serialize(
        Archive & ar,
        T & t,
        const unsigned int file_version
    ){
        ...
    }

    template<class Archive, class T>
    void serialize(
        Archive & ar,
        my_template<T> & t,
        const unsigned int file_version
    ){
        ...
    }

The serialization library works around this issue by using a different default definition of the first template:

    template<class Archive, class T>
    void serialize(
        Archive & ar,
        T & t,
        const unsigned long int file_version  // Note: change to long
    ){
        ...
    }

Now, the second template is not matched with the first one so there is no PFTO and no compile error. When the serialization library invokes

    serialize(ar, t, 0);

the function declaration is first matched against templates with an integer for the third argument. If there is a match, the matching template is instantiated and later invoked. If there is no match, an attempt is made to match other templates by converting arguments to other types. In this case the third argument can be converted to long to match the first template - which is the default. So in this case, the first template will be instantiated and later invoked. We have managed to use function overloading to achieve the same effect as PFTO were it correctly implemented.

This depends upon undefined behavior of a compiler already determined to be non-conforming. In other words, there is no guarantee that this will work on all compilers. If a compiler does not correctly support PFTO and this method cannot be used to workaround it, non-intrusive serialization cannot be supported for that compiler. As of this writing, such a compiler has not been encountered.

It turns out that using this "trick" can create problems with compilers that DO correctly support PFTO. For this reason we define a macro `BOOST_PTFO` which is defined to be `long` for non-conforming compilers and nothing for conforming ones. So the default definition is really: The serialization library works around this issue by using a different default definition of the first template:

    template<class Archive, class T>
    void serialize(
        Archive & ar,
        T & t,
        const unsigned BOOST_PFTO int file_version  // Note: change to BOOST_PFTO
    ){
        ...
    }

### Character Encoding

The whole question of character encoding combined with wide characters is much more complicated than it would seem to be. The current library defines in 3 formats (text, binary, and XML), wide and narrow characters, an attempts to be portable between compiler libraries. The results of a rather long consideration of all these factors has been to set default encoding according to the following rules.

- All text archives (i.e. `text_?archive`) will produce text output in the current stream `locale`. Generally this will produce no changes in string data.
- To produce binary output with Microsoft compilers, the stream will have to be opened with mode `ios::binary`. Failure to do so will result in 0x0d characters (carriage-return) characters being removed from the input stream if they are followed by a 0x0a character (line-feed). This could corrupt the input and make the file unreadable. On UNIX systems the `ios::binary` is not required and is ignored if used.
- character XML archives (i.e. xml_oarchive) will produce XML output with characters encoded according to the current stream `locale`.
- wide character XML archives (i.e. xml_woarchive) will produce files encoded in UTF-8.

This character encoding is implemented by changing the `locale` of the i/o stream used by an archive when the archive is constructed, the stream local is changed back to its original value. This action can be overridden by specifying `boost::archive::no_codecvt` when the archive is opened. In this case, the stream `locale` will not be changed by the serialization library.

Note that the code conversion included for wide character text and XML archives could alter `std::string` data stored in archives. Suppose a normal (multi-byte) character string is written to a wide character stream. Our system uses the current `locale` to translate it to a wide character string before writing it out. Upon reading, it is translated back to a (multi-byte)string. If the `locale` on the platform that reads the archive is different than the `locale` on the platform that wrote the stream, the actual string data may be altered by the serialization process. To avoid this, either avoid usage of `locale` dependent multi-byte strings or be sure that the `locale` is set correctly before reading the archive.

To produce wide character text output (i.e. 16 bit characters on Win32 systems), do the following.

- Open a wide character stream.
- Alter the stream `locale` to use `boost::archive::codecvt_null<OStream::char_type>`
- Create the archive with the flag `no_codecvt`.

Naturally, the input process has to be symmetrical.

### Partial Template Specialization

Compilers which fail to support partial template specialization will fail to compile the following code. To make this compiler, the `const` has to be removed.

    void f(A const* a, text_oarchive& oa)
    {
      oa << a;
    }

### Template Invocation syntax

Some compilers may not recognize the syntax:

    ar.template register_type<T>();

for "registering" derived pointers of polymorphic classes. The actual function prototype is:

    template<T>
    void register_type(T * t = NULL);

so that one may write `ar.register_type(static_cast<T *>(NULL))` instead of the syntax described above.

### Specific Compiler/Library Issues

#### GCC 3.X

GCC versions for Cygwin and MinGW fail to support wide character I/O. So all tests using wide char I/O fail. Note that if wide character I/O support is added with STLPort, all tests complete successfully.

#### GCC 2.95

All of the above plus:

- The serialization library depends on the templeted stream implemention to function properly. So STLPort must be used to build the library.
- Polymorphic archive tests fail.

#### Intel C++ 8.0

No known issues. All tests compile and run in debug and release modes.

#### Visual C++ 7.1

No known issues. All tests compile and run in debug and release modes.

#### Visual C++ 7.0

- The "pimpl" demo fails to link. Cause and workaround for this is unknown
- XML serialization only works with version 1.6x of spirit. In order to build and use this library with this compiler, one must use version 1.6x rather than the latest version shipped with boost. See Release Notes.

#### Visual C++ 6.0

all the above issues for Visual C++ 7.0 plus:

- Out of line template definitions are not recognized and fail with a confusing error message. To function save/load/serialize member function templates must be defined within the class definition. This feature is essential to `demo_pimpl`. Hence, this program will fail to compile. In this case the problem can't be worked around and still demonstrate this facility.
- This compiler does not support `wchar_t` as a separate type. It defines `wchar_t` as an alias for `short int`. In general things will still function. However certain customization, such as overloading archive operators for saving/loading wide character arrays would produce surprises in this environment.
- Under certain circumstances, a program will fail to link with the message: LIN1179 - "invalid or corrupt file: duplicate comdat". According to http://groups.google.com/groups?th=8a05c82c4ffee280 (look for P78) A LNK1179 error occurs when:
  - The template class takes at least two arguments.
  - The template is used at least two times with identical first and different second arguments.
  - The static member variable is of an object type with at least one base class. (In another scenario it also occurred using a member without a base class.)
  Working around this in the implementation of the library for this compiler entailed a ridiculous amount of effort. Even so, the effort wasn't entirely succesful. With this compiler, this message will still appear under the following conditions:
  - When serializing a class with multiple base classes. This problem causes two failure in the test suite. I have been unable to divise a way to work around this.
  - Using more than one kind of archive in the same code module. This should be easy to work around in practice.
- Code modules exceeding some undetermined size that use the library will fail with fatal error C1204: compiler limit : internal structure overflow. This can be addressed by dividing the module into smaller ones.

#### Borland 5.64

- `enum` data members cannot be serialized. Conversion to/from integers will work around the problem.
- Default array serialization fails. Workaround this by doing it with a loop.
- If class serialize functions are not accessable either by making them public or by including `friend` declarations as described in Class Serialization - Member Function, the will compile but fail at runtime.
- tests using custom extended type which doesn't use rtti fails.

#### Borland 5.51 and earlier

All of the above issues for Borland 5.64 plus:

- Most tests using Wide character XML files fail. This happens somewhere within the spirit library but we've been unable to track it further than this.
- A couple of other tests fail.

#### Dinkumware Library

Several compilers, including Visual C++ 6.0, use an older dinkumware library. These platforms have several issues:

- The dinkumware library shipped with this compiler does not change the locale facet of an i/o stream unless the `imbue` function is called before the the stream is opened. In order to use this library with this environment to generate UTF-8 files, one cannot depend on the "automatic" setting of local that archives implement. The stream local must be set explicitly on the stream before an archive is opened on it. The archive should be opened with the `no_codecvt` flag. Note this problem will occur on all compilers shipped with this library.
- Other issues have been worked around in the file. dinkumware.hpp

Revised 1 November, 2004

© Copyright Robert Ramey 2002-2004. Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)