serialiazation, carchive issue with vc8 and vc6

Hi all

I've recently exported one of our tools to VC8, after some minor
changes it compiles and appears to work ok, it can open and close files
which it has created etc. The vc8 migrated app exports a file using
carchive which again appears to be workign fine except when it is to be
read using an application compiled using vc6.
Are there any known problems when reading vc8 serialized files with a
vc6 product.
Note that migrating the vc6 project to vc8 is currently not an option

David
Thu Jul 20 14:55:33 CDT 2006

>The vc8 migrated app exports a file using
>carchive which again appears to be workign fine except when it is to be
>read using an application compiled using vc6.

What happens if you read an archive file from the VC6 built
application?

Have you tried debugging the VC6 built application to identify the
cause of the problem? ... i.e. is it in some of your code, or some
standard MFC class?

Dave

audrey
Fri Jul 21 03:20:25 CDT 2006

I've narrowed it down to the serialization of a CTime object

if i remove the CTime object all the problems go away
I picked up on a tiny thread of info yesterday that there is possibly a
problem with CTime in VC* but havn't found sufficient information to
find a fix


David Lowndes wrote:

> >The vc8 migrated app exports a file using
> >carchive which again appears to be workign fine except when it is to be
> >read using an application compiled using vc6.
>
> What happens if you read an archive file from the VC6 built
> application?
>
> Have you tried debugging the VC6 built application to identify the
> cause of the problem? ... i.e. is it in some of your code, or some
> standard MFC class?
>
> Dave

David
Fri Jul 21 06:09:22 CDT 2006

>I've narrowed it down to the serialization of a CTime object

Is this a 32-bit CTime vs. 64-bit CTime issue perhaps?

Dave

audrey
Fri Jul 21 07:41:41 CDT 2006


that sounds promising, tell me more

i assume you mean that the vc8 serialization uses 64bit and i'm tryign
to read it with the vc6 32bit version?



David Lowndes wrote:
> >I've narrowed it down to the serialization of a CTime object
>
> Is this a 32-bit CTime vs. 64-bit CTime issue perhaps?
>
> Dave

David
Fri Jul 21 09:56:41 CDT 2006

>i assume you mean that the vc8 serialization uses 64bit and i'm tryign
>to read it with the vc6 32bit version?

Possibly. There's a macro definition in VC8 to have it use the 32-bit
time - _USE_32BIT_TIME_T. I don't know if that is your problem or
whether that will workaround the issue though, it's just a guess.

Dave

Alex
Sun Jul 23 02:48:26 CDT 2006

"David Lowndes" wrote:
> >i assume you mean that the vc8 serialization uses 64bit
> >and i'm tryign
>>to read it with the vc6 32bit version?
>
> Possibly. There's a macro definition in VC8 to have it use
> the 32-bit
> time - _USE_32BIT_TIME_T. I don't know if that is your
> problem or
> whether that will workaround the issue though, it's just a
> guess.


Even if will work, it's a good lesson to original poster to
avoid binary serialization in the future.

mnb111
Tue Jul 25 18:20:02 CDT 2006

Regarding your comment about "binary serialzation", for those of us coming
from VS6, we didn't have many options other that MFC's CArchive. We have a
very extensive app based on VS6 C++. Am I missing something here or was your
suggestion refering to "starting from scratch"?

We obviously ran into the same problem with CTime. This was well over a year
ago that we did the first port (first and only). We searched high and low for
any info about CTime and came up empty. Where did you find the info on the
macro and is it something new?

What we did was to deserialize into a temp and convert and reserialize with
a new version. Is there any MFC or VS support for this type of migration?

Just wondering as we've never heard of the macro (even though it makes no
sense to us to keep CTime as 32-bit). Thanks for any info you can provide.
--

-mike


"Alex Blekhman" wrote:

> "David Lowndes" wrote:
> > >i assume you mean that the vc8 serialization uses 64bit
> > >and i'm tryign
> >>to read it with the vc6 32bit version?
> >
> > Possibly. There's a macro definition in VC8 to have it use
> > the 32-bit
> > time - _USE_32BIT_TIME_T. I don't know if that is your
> > problem or
> > whether that will workaround the issue though, it's just a
> > guess.
>
>
> Even if will work, it's a good lesson to original poster to
> avoid binary serialization in the future.
>
>
>

Carl
Tue Jul 25 18:47:32 CDT 2006

"mnb111" <mnb111@newsgroups.nospam> wrote in message
news:EA1836EE-6A4C-4AFE-9D7B-E370B32E406B@microsoft.com...
> Regarding your comment about "binary serialzation", for those of us coming
> from VS6, we didn't have many options other that MFC's CArchive. We have a
> very extensive app based on VS6 C++. Am I missing something here or was
> your
> suggestion refering to "starting from scratch"?
>
> We obviously ran into the same problem with CTime. This was well over a
> year
> ago that we did the first port (first and only). We searched high and low
> for
> any info about CTime and came up empty. Where did you find the info on the
> macro and is it something new?
>
> What we did was to deserialize into a temp and convert and reserialize
> with
> a new version. Is there any MFC or VS support for this type of migration?
>
> Just wondering as we've never heard of the macro (even though it makes no
> sense to us to keep CTime as 32-bit). Thanks for any info you can provide.

The macro is new with VC8, and is documented (somewhat) at:

http://msdn2.microsoft.com/en-us/library/ms235429.aspx

That CTime is just a wrapper around a time_t isn't clearly documented, that
I can find. You might infer from the inconsistencies in the documentation
of CTime::GetTime() at

http://msdn.microsoft.com/library/en-us/vclib/html/_mfc_ctime.3a3a.gettime.asp?frame=true

that this is indeed the case - the function is documented as returning
__time64_t, but the example shows it returning time_t.

It's a wart, no doubt about it. That is a danger with binary
serialization - unless you explcitly control the layout of everything in the
archive, you're subject to version incompatibilities.

-cd

mnb111
Tue Jul 25 22:31:02 CDT 2006

Thanks for the reply Carl, I was just "sanity checking" to make sure I hadn't
missed anything. The reason for binary serialization is the nature of our
app. I won't bore you with all the gory details, but we live in the world of
critical 24x7 hi-performance apps. We serialize 250,000 objects at the bottom
of the tree and it's crucial that we can persist the information without
impacting the other threads. Synchronization is highly optimized so we don't
lock anyone out for longer than absolutely necessary. It's not unusual for
our app to go for 6 months without being bounced (Windows has come a long
way).

Anyways, we reconfigure the app on the fly, change database schemas, and the
only way anyone besides the person making the changes even knows something
has happened is if they check the logfile and notice their connection was
bounced - normally for less than a second.

I'm always on the lookout for better ways of doing things - these days no
one can keep up on all the latest technology.
--

-mike


"Carl Daniel [VC++ MVP]" wrote:

... snipped [refer to the thread if interested] - mnb

>
> It's a wart, no doubt about it. That is a danger with binary
> serialization - unless you explcitly control the layout of everything in the
> archive, you're subject to version incompatibilities.
>
> -cd
>
>
>

Alex
Wed Jul 26 03:53:36 CDT 2006

"mnb111" wrote:
> Regarding your comment about "binary serialzation", for
> those of us coming
> from VS6, we didn't have many options other that MFC's
> CArchive. We have a
> very extensive app based on VS6 C++. Am I missing
> something here or was your
> suggestion refering to "starting from scratch"?
>
> We obviously ran into the same problem with CTime. This
> was well over a year
> ago that we did the first port (first and only). We
> searched high and low for
> any info about CTime and came up empty. Where did you find
> the info on the
> macro and is it something new?
>
> What we did was to deserialize into a temp and convert and
> reserialize with
> a new version. Is there any MFC or VS support for this
> type of migration?
>
> Just wondering as we've never heard of the macro (even
> though it makes no
> sense to us to keep CTime as 32-bit). Thanks for any info
> you can provide.


I always considered CArchive rather as proof of concept,
than solution for real life product. It's good for
small/middle sized application as quick and dirty solution.
But when you need to maintain several versions with backward
compatibility for previously saved files, then all hell
breaks loose.

I read your another response to Carl Daniel where you
describe your serialization requirements. In this case, when
speed of save/load is paramount, then binary format can be
the choice. I never measured serialization times personally,
and it can be quite interesting to compare textual
serialization speeds against binary ones (including
workarounds for previous versions for binary).

As possible alternative for raw binary serialization there
is Structured Storage, which implements kind of filesystem
within a file. It can be reasonable compromise between
binary storage and backward compatibility issues.

HTH
Alex

Carl
Wed Jul 26 09:03:31 CDT 2006

Alex Blekhman wrote:
> As possible alternative for raw binary serialization there
> is Structured Storage, which implements kind of filesystem
> within a file. It can be reasonable compromise between
> binary storage and backward compatibility issues.

...it just has one big wart: it's terribly, terribly slow. Or at least it
was in the past. I recall that NTFS5 (aka Windows XP) has some sort of
native support for structured storage in the filesystem, so it's possible
that it's not so horribly slow anymore.

-cd

Alex
Wed Jul 26 09:24:03 CDT 2006

"Carl Daniel [VC++ MVP]" wrote:
>> As possible alternative for raw binary serialization
>> there
>> is Structured Storage, which implements kind of
>> filesystem
>> within a file. It can be reasonable compromise between
>> binary storage and backward compatibility issues.
>
> ...it just has one big wart: it's terribly, terribly
> slow. Or at least it was in the past. I recall that
> NTFS5 (aka Windows XP) has some sort of native support for
> structured storage in the filesystem, so it's possible
> that it's not so horribly slow anymore.

It can be for sure. I heard somewhere that Registry files
are implemented as structural storage actually. I can be
wrong about this. However, if it is true, then structured
storage better be fast.

Carl
Wed Jul 26 10:03:32 CDT 2006

Alex Blekhman wrote:
> "Carl Daniel [VC++ MVP]" wrote:
>>> As possible alternative for raw binary serialization
>>> there
>>> is Structured Storage, which implements kind of
>>> filesystem
>>> within a file. It can be reasonable compromise between
>>> binary storage and backward compatibility issues.
>>
>> ...it just has one big wart: it's terribly, terribly
>> slow. Or at least it was in the past. I recall that
>> NTFS5 (aka Windows XP) has some sort of native support for
>> structured storage in the filesystem, so it's possible
>> that it's not so horribly slow anymore.
>
> It can be for sure. I heard somewhere that Registry files
> are implemented as structural storage actually. I can be
> wrong about this. However, if it is true, then structured
> storage better be fast.

No, registry hives are not structured storage, although there are
similarities.

-cd

Doug
Wed Jul 26 10:06:16 CDT 2006

On Wed, 26 Jul 2006 07:03:31 -0700, "Carl Daniel [VC++ MVP]"
<cpdaniel_remove_this_and_nospam@mvps.org.nospam> wrote:

>...it just has one big wart: it's terribly, terribly slow. Or at least it
>was in the past. I recall that NTFS5 (aka Windows XP) has some sort of
>native support for structured storage in the filesystem, so it's possible
>that it's not so horribly slow anymore.

I dunno, ISTR that Native Structured Storage, which implemented structured
storage in terms of NTFS alternate streams, was introduced but later
withdrawn sometime during the Win2K beta.

--
Doug Harrison
Visual C++ MVP

audrey
Wed Jul 26 10:53:56 CDT 2006

well cheers peeps

i've made a fix which will suffice until our reading app is migrated to
vc8
i've simply taken the time in seconds returned from time_t::time() and
cast it into an __int32 before serializing it.
Plugs a gap for now..

toodle pips

audrey

Doug Harrison [MVP] wrote:
> On Wed, 26 Jul 2006 07:03:31 -0700, "Carl Daniel [VC++ MVP]"
> <cpdaniel_remove_this_and_nospam@mvps.org.nospam> wrote:
>
> >...it just has one big wart: it's terribly, terribly slow. Or at least it
> >was in the past. I recall that NTFS5 (aka Windows XP) has some sort of
> >native support for structured storage in the filesystem, so it's possible
> >that it's not so horribly slow anymore.
>
> I dunno, ISTR that Native Structured Storage, which implemented structured
> storage in terms of NTFS alternate streams, was introduced but later
> withdrawn sometime during the Win2K beta.
>
> --
> Doug Harrison
> Visual C++ MVP

mnb111
Wed Jul 26 11:48:01 CDT 2006

I don't know about "proof of concept", but like most wrappers (I consider
anything that ends up in the SDK, a wrapper), you have to know what its
capabilities are before you use it. Obviously, this only refers to apps where
performance is an issue (reliablilty is a given - I can't imagine writting
any app other than a prototype where I wouldn't be concerned about
robustness).

As it turns out, we had to bypass serialize in some of the collection
classes. When we were profiling, we found an inordinate amount of time being
spent in some of them. It's pretty simple if speed is important. Serialize
the version (every object that's serialized has a version up front), the
count, and loop while adding the necessary members.

We didn't start this way, but every major release has at least doubled the
capacity. The last release went from supporting around 20,000 items to
250,000. It was an interesting exercise. What was tolerable at 20K was dismal
at 250K. Only then did we resort to tatics such as bit-fields for boolean
members and the bypassing of some of the serialization methods. At the leaf
elements, we would serialize a version then the collection as we couldn't
afford the overhead of a version in the object.

Bottom line is as long as you're careful and pay attention (of course you
have to know what you're doing), you can push Windows and MFC/Visual Studio
way beyond what most people would imagine. The 250K item server was running
on a Dell Workstation 530 (dual 1.7GHz Xeon) and archiving 250K+ items a
minute, and processing 18,000 TCP packets/sec (2 gigabit NICs in a team).
Believe it or not, the limit we hit was the TCP stack (we actually test the
limits so we know where they are). This server had around 120 threads and
contrary to popular belief, that many threads is not only feasible, they
server a purpose.

Pardon my rambling, seem to be a long way off the subject of migrating to
64-bit CTime. Just pointing out why some of us have to get "dirty". I still
have a group that thinks they can create a high performance app in .NET using
Web Services and remoting. Have you ever looked into what happens with a Web
Service call?

Don't get me wrong, .NET has its place as well as Web Services. Just use the
right tool for the job - of course this means knowing the capabilities of the
tool.
--

-mike


"Alex Blekhman" wrote:

> "mnb111" wrote:
> > Regarding your comment about "binary serialzation", for
> > those of us coming
> > from VS6, we didn't have many options other that MFC's
> > CArchive. We have a
> > very extensive app based on VS6 C++. Am I missing
> > something here or was your
> > suggestion refering to "starting from scratch"?
> >
> > We obviously ran into the same problem with CTime. This
> > was well over a year
> > ago that we did the first port (first and only). We
> > searched high and low for
> > any info about CTime and came up empty. Where did you find
> > the info on the
> > macro and is it something new?
> >
> > What we did was to deserialize into a temp and convert and
> > reserialize with
> > a new version. Is there any MFC or VS support for this
> > type of migration?
> >
> > Just wondering as we've never heard of the macro (even
> > though it makes no
> > sense to us to keep CTime as 32-bit). Thanks for any info
> > you can provide.
>
>
> I always considered CArchive rather as proof of concept,
> than solution for real life product. It's good for
> small/middle sized application as quick and dirty solution.
> But when you need to maintain several versions with backward
> compatibility for previously saved files, then all hell
> breaks loose.
>
> I read your another response to Carl Daniel where you
> describe your serialization requirements. In this case, when
> speed of save/load is paramount, then binary format can be
> the choice. I never measured serialization times personally,
> and it can be quite interesting to compare textual
> serialization speeds against binary ones (including
> workarounds for previous versions for binary).
>
> As possible alternative for raw binary serialization there
> is Structured Storage, which implements kind of filesystem
> within a file. It can be reasonable compromise between
> binary storage and backward compatibility issues.
>
> HTH
> Alex
>
>
>

Alex
Thu Jul 27 03:34:09 CDT 2006

"mnb111" wrote:
> I don't know about "proof of concept", but like most
> wrappers (I consider
> anything that ends up in the SDK, a wrapper), you have to
> know what its
> capabilities are before you use it. Obviously, this only
> refers to apps where
> performance is an issue (reliablilty is a given - I can't
> imagine writting
> any app other than a prototype where I wouldn't be
> concerned about
> robustness).
>
> As it turns out, we had to bypass serialize in some of the
> collection
> classes. When we were profiling, we found an inordinate
> amount of time being
> spent in some of them. It's pretty simple if speed is
> important. Serialize
> the version (every object that's serialized has a version
> up front), the
> count, and loop while adding the necessary members.
>
> We didn't start this way, but every major release has at
> least doubled the
> capacity. The last release went from supporting around
> 20,000 items to
> 250,000. It was an interesting exercise. What was
> tolerable at 20K was dismal
> at 250K. Only then did we resort to tatics such as
> bit-fields for boolean
> members and the bypassing of some of the serialization
> methods. At the leaf
> elements, we would serialize a version then the collection
> as we couldn't
> afford the overhead of a version in the object.
>
> Bottom line is as long as you're careful and pay attention
> (of course you
> have to know what you're doing), you can push Windows and
> MFC/Visual Studio
> way beyond what most people would imagine. The 250K item
> server was running
> on a Dell Workstation 530 (dual 1.7GHz Xeon) and archiving
> 250K+ items a
> minute, and processing 18,000 TCP packets/sec (2 gigabit
> NICs in a team).
> Believe it or not, the limit we hit was the TCP stack (we
> actually test the
> limits so we know where they are). This server had around
> 120 threads and
> contrary to popular belief, that many threads is not only
> feasible, they
> server a purpose.
>
> Pardon my rambling, seem to be a long way off the subject
> of migrating to
> 64-bit CTime. Just pointing out why some of us have to get
> "dirty". I still
> have a group that thinks they can create a high
> performance app in .NET using
> Web Services and remoting. Have you ever looked into what
> happens with a Web
> Service call?
>
> Don't get me wrong, .NET has its place as well as Web
> Services. Just use the
> right tool for the job - of course this means knowing the
> capabilities of the
> tool.


I agree about right tool for the job. I don't know your
product well enough to suggest something more specific than
general common sense things. Probably you have the
situation when binary serialization pain is justified
because of the speed it gains.

I think you remember well the formats war in beginning and
mid-90's. The majority of general purpose applications
didn't benefit from it. On the contrary, valuable resources
were spent on compatibility issues, conversion from
competitors formats and plain obfuscation of own formats.
All this happened at expense of useful features and/or
stability. Thanks god, today software industry started to
sober and tries to develop some standards, so different
applications would be able to interact with each other.