requests for files in a archive are made with unix style paths. So to
be able to match an entry in map(_indexMap) it's keys needs to be
stored in unix style even on Win32"
Note from Robert Osfied, simplified this submission so that the added conversion to
unix slahes is done on all platforms as this should be safe and simpler to maintain.
64 bit binary compatible OSGA archive reader/writer using mixed
stdio/iostream calls. But during this work I learned that it can be made in
much simpler way.
Attached is result of this new attempt. I hope its appropriate for inclusion
into OSG codebase. It was compiled and tested with latest SVN OSG, Windows
XP 32 bit and Windows Vista business 64 bit. OSG was built using VS 2005
Express SP1 for 32 bit environment and VS 2005 Std for 64 bit.
---
Solution description (there were two problems involved):
---
Problem 1: implicit conversions beetween file positions and 32 bit int. This
could be considered a MS compiler bug because this 32 bit int was
additionally implicitly converted to/from 64 bit. As far as I know compiler
is allowed to make only one implict conversion (but maybe this rule does not
refer to simple types).
Its actually possible to address OSGA files above 4 GiB range using 32 bit
windows iostreams. MS Iostreams in practice offer the same level of
functionality as stdio functions. There are functions fsetpos and fgetpos in
stdio lib which use 64 bit file pointers (fpos_t). These functions are
internally called by seekp( streampos ), seekg( streampos ), tellp(), and
tellg() methods. So its also possible to change and retrieve file postions
using iostream calls. But the problem lies in implicit handling of streampos
type.
streampos type is actually a template class used as seekp, seekg parameter
and returnd from tellp, tellg. Its capable of storing 64 bit file pointers.
But streampos can be also converted to/from simple type streamoff. It has
proper constructor and cast operator. In Win 32 environment streamoff is
defined as long (~32 bit int). So when seekp, and tellp arent used with
exact streampos objects but OSGA_Archive::pos_type complier makes implicit
casts to 32 bit int types loosing important bits of information.
So above problem could be easily handled by making conversion calls
explicit. My code defines 2 functions used to convert back and forth beetwen
64 bit OSGA_Archive::pos_type and std::streampos objects:
OSGA_Archive::pos_type ARCHIVE_POS( const std::streampos & pos );
std::streampos STREAM_POS( OSGA_Archive::pos_type & pos );
Rest of the OSGA implementation code was modified to call these conversions
explicitly with seekp, seekg, tellp, tellg.
---
Problem 2: seekp and seekg have two variants. Only one of these variants is
actually 64 bit proof.
When I solved my first problem and made use of explicit streampos conversion
functions, OSGA archive was able to read my example 11 GiB archive. But
there were still problems with write and append. I found that the reason for
this was pair of seekp( 0, std::ios_base::end ) and tellp() calls. It turned
out that use of seekp, seekg( offset, direction ) function variants was
setting file pos pointer to EOF when file was larger than 4GiB. But I
noticed that one arg seekp, seekg ( streampos ) versions worked correctly.
So the solution was to change OSGA write logic a little, and replace
seekp( offset, direction ) with seekp( absolute_pos ) calls.
I achieved this by modifing IndexBlock write method to record and restore
file pos after IndexBlock was written. This modification has the effect that
put pointer is generally kept at the end of file, so there is no need to
repostion to the end before writing the files. This allowed me to get rid of
those problematic seekp( 0, std::ios_base::end ) calls.
There was one place where I could not easily get rid of seekp( 0,
std::ios_base::end ). It was situation where existing OSGA was opened for
appending. I resolved this by computing file length by finding max position
from index block and file block endings. Then I replaced former seekp( 0,
std::ios_base::end ) with seekp( STREAM_POS( found_file_length ).
---
Description of these changes may sound bit hacky but in practice these were
fairly simple and straightforward modifications. I hope they pass your
review. There is one complex preprocessor condition which I based on few
lines taken from boost positioning.hpp. Boost licence does allow such
reproduction. In case of problems this condition may be easily simplified to
windows only implementation.
"
"On fixing the pointer access I discovered that reading osga archives
containing ive files went into a cpu loop. This turned out to be a
problem with proxy_streambuf on Solaris. Public methods in the Solaris
streambuf standard library implementation rely on the gptr() being set,
which proxy_streambuf was not doing. So I have modified
proxy_streambuf to set the input sequence pointers, and have also
aligned it more with the standard library streambuf implementation
where all input is through underflow(), not uflow() which merely calls
underflow() and advances the pointer."
From Robert Osfield, change from using pointer cast and assignment to using
a templated _write and _read method to avoid pointer aliasing to 2/4/8
byte boundaries that some computer system may produce. These changes
where inspried by Colin McDonalds change to using memcpy, these
changes weren't merged as memcpy is not as clear in naming as _read,
_write and memcpy will incurr a function call just for copy a
uint.
